Students’ motives for using online formative assessments when preparing for summative assessments

Abstract

Background: Formative assessments intend to provide feedback on student performance in order to improve and accelerate learning. Several studies have indicated that students using online formative assessments (OFAs), have better results on their exams.

Aims: The present study aims to provide insight in student reasons for using or not using available OFAs.

Method: Three OFAs with feedback were available in a second year undergraduate course in physiology for biomedical sciences students (N = 147). First, students received an open questionnaire about why they did (not) complete the first two OFAs. Based on this data, a closed questionnaire was developed and distributed among students. Exploratory factor analysis (EFA) was applied.

Results: The results indicate reasons why students do (not) use the OFAs. The EFA for using the OFAs indicated three factors, that were interpreted as collecting (1) feed up, (2) feed forward, and (3) feed back information. The main reasons for not using the OFAs were lack of time and having completed the questions before.

Conclusions: Students’ reasons for using OFAs can be described in terms of collecting feed up, forward and back information and students’ reasons for not using OFAs can be student-, teacher-, or mode-related.

Introduction

Self-directed learning is one of the most promising types of learning advocated in medical education to prepare medical students for continuing professional education (Collins & Hammond 1987; Candy 1995; Davis et al. 1995; Murad et al. 2010). Self-directed learners are able to self-appraise their work and to seek, accept and use feedback from others in order to improve their performance (Sargeant et al. 2008). A tool that can be used to support the self-directed learning of students is an online formative assessment (OFA) that students can voluntarily take part in while preparing for a summative exam (Henley 2003; Kibble 2007; Gikandi et al. 2011; Wang 2011). Formative assessment refers to an assessment that is specifically intended to provide feedback on performance to improve and accelerate learning (Sadler 1998; Rushton 2005), as opposed to a summative assessment that summarises the achievement of a student, often in the form of a grade (Sadler 1989). The results from a formative assessment have value not in terms of completing a course, but rather providing feedback to the students as to the extent of their understanding of the course material. Thus, it can help them in planning their next learning activities (Carrillo-de-la-Pena et al. 2009).

Several studies have investigated the effects of OFAs (e.g. Buchanan 2000; Olsen & McDonald 2004; Kibble 2007; Dobson 2008; Velan et al. 2008; Angus & Watson 2009; Kibble et al. 2011; Velan et al. 2002; Bouwmeester et al. accepted), all indicating that students participating in the OFA had higher scores on the subsequent summative assessment. In order to explain this effect of OFAs, several mechanisms have been proposed: increasing student engagement (Peat & Franklin 2002; Gikandi et al. 2011); increasing time on task (Cook et al. 2010); preventing procrastination (Peat & Franklin 2002); and providing formative and informative feedback (Velan et al. 2002; Carrillo-de-la-Pena et al. 2009; Gikandi et al. 2011). Interestingly however, despite the positive effects of OFAs, not all students use them (Sly 1999; Kibble 2007). This raises the issue of why students do or do not use OFAs and how these student reasons might be related to the different mechanisms that are proposed and so far little is known about this issue (Sinclair & Cleland 2007). Therefore, the present study seeks to better understand the effect of OFAs by asking students why they chose to use or not use OFAs and by addressing the following research question: What reasons do students have for using online formative assessments?

Table

Practice points

Student motives for using OFAs are collecting feed up, feed back, and feed forward Not all students student use OFAs Student motives for not using OFAs can be teacher-, student-, and mode-related When designing OFAs, constructive alignment should be borne in mind

Insight into students’ reasons for using OFAs can (a) contribute to our understanding of the mechanisms that explain the relation between participation in OFAs and higher scores on summative assessments, (b) establish guidelines for the design and implementation of OFAs in medical education that are aligned with student reasons for using them and (c) provide suggestions for making OFAs more appealing to students who do not use them instantly.

Methods

This study was conducted in a second year undergraduate course in physiology for biomedical sciences students. In total, 147 students took the course, which was made up of three blocks in which the respiratory, circulatory and urinary systems were addressed. At the end of each block a summative multiple choice test was taken by the students, assessing their knowledge and skills with respect to the specific organ system. In addition, at the end of the whole course a final summative test with essay questions was taken by the students in which knowledge and skills of the three organ systems were tested.

For each of the three organ systems OFAs were available in the e-learning environment, to which all students had access. Students received feedback on their answers when they had finished the complete assessment. In case of a correct answer, the feedback confirmed/elaborated why the answer was correct. In case of a wrong answer, the feedback indicated what mistake the student most likely made. The questions of the OFA were also available beneath micro lectures of the lectures, so that students could make the test items right after watching the micro lectures. If they did so, this was not considered using the complete OFA.

After the summative assessment of the second block, using tracking data from the online learning environment, we divided the students into four groups: (a) students who completed both OFA 1 (respiratory system) and OFA 2 (circulatory system) (N = 88), (b) students who completed only OFA 1 (N = 15), (c) students who completed only OFA 2 (N = 20) and (d) students who completed neither OFA 1 nor OFA 2 (N = 24). Each group received a specific email with a link to an online questionnaire with open questions about the formative assessments, so that the questions could be adapted to whether or not they completed one or two of the available OFAs (see Table 1). Respectively 43, 6, 7 and 4 students of the four groups completed the questionnaire, indicating a general response rate of 40.8%. More specifically, the response rates for the four groups of students were 49% for group (a), 40% for group (b), 35% for group (c), and 17% for group (d). This indicates a trend of students that used the OFAs being more likely to fill out the questionnaire. Two independent researchers deductively constructed a coding scheme for the short answers. Both coding schemes were compared and differences between the schemes were discussed until consensus was reached (Creswell & Miller 2000). This led to the final coding scheme presented in Table 2. All student answers were coded (Cohen's kappa for reasons for completion: 0.76; Cohen's kappa for reasons for non-completion: 0.78).

Table 1.  Questions of the first student questionnaire

Completed online formative assessment	Scale
1. What were your main reasons for completing OFA 1/OFA 2?	Short answer
2. Completing OFA 1/OFA 2 had an added value in preparing for the summative assessment.	5 point scale (agree – disagree)
If agree: What was the added value?	Short answer
Not completed online formative assessment
3. What were your main reasons for not completing OFA 1/OFA 2?	Short answer
General
4a. Are you planning on completing OFA 3?	Yes/No
4b. What are your main reasons for this?	Short answer

Table 2.  Final coding scheme for reasons for completing and not completing OFA

Reasons for completing online formative assessment

1. Checking whether I studied sufficiently

2. Repeating and rehearsing the course material

3. Getting insight into the form and content of the summative assessment (including being prepared for the possible use of the same test items)

4. Discovering gaps in my knowledge and skills

5. Understanding what elements of the course material are important

6. Having a positive/negative experience with the previous formative/summative assessment

7. Other

Reasons for not completing online formative assessment

1. Having too little time/completing the online formative assessments takes too much time

2. Having completed the test items below the micro lectures already

3. Already knowing the course material well enough

4. Finding the online formative assessment insufficiently representative of the summative assessment

5. Experiencing technical problems

6. Preferring to learn from other sources

7. Forgetting the possibility of making the online formative assessment

8. Other

Based on the final coding scheme a questionnaire was developed containing 32 questions that had to be answered on a five-point scale. In order to establish a higher response rate than was established for the online questionnaire with open questions, this questionnaire was administered on paper right after the students took the summative assessment of the third block (urinary system). So, the students could fill out the questionnaire after completing the summative assessment and hand in both at the same time. In total, 134 students returned the completed questionnaire, indeed indicating a good response rate of 91.1%. These data were analysed by inspection of the means and standard deviations and factor analyses in order to explore whether latent factors underlying students’ reasons for completing and not completing the OFAs could be found. For the factor analyses, following the advice of Costello and Osborne (2005), we used maximum likelihood estimation in combination with a direct Oblimin rotation. The number of factors was determined by inspecting the scree plot.

Results

Reasons for completing the OFAs

Most variables violated the assumption of a normal distribution, and all variables had a median of 4. Therefore, also Table 3 also presents the means and standard deviations for all variables. The fact that the median for all variables is 4, indicates that all reasons play a role for students. The first two reasons (“checking whether I studied sufficiently” and “repeating and rehearsing the course material”) proved the most important reasons for students to complete the OFA, followed by “discovering gaps in my knowledge and skills.” In addition, the least important for students were “understanding what elements of the course material are important” and “having a positive/negative experience with the previous formative/summative assessment.”

Table 3.  Means and SDs for reasons for completing OFAs and the added value in terms of means and standard deviations (N = 104)

Reasons	To complete OFA 3 (retrospect)	For added value of OFA 3 (retrospect)	To complete OFAs before the final test (prospect)
Checking whether I had studied sufficiently	4.3 (0.7)	4.0 (0.6)	4.1 (0.7)
Repeating and rehearsing the course material	4.3 (0.6)	4.1 (0.6)	4.4 (0.6)
Getting insight into the form and content of the summative assessment	4.0 (0.8)	3.6 (0.9)	3.6 (1.1)
Discovering gaps in my knowledge and skills	4.2 (0.8)	3.9 (0.9)	4.1 (0.8)
Understanding what elements of the course material are important	3.6 (1.1)	3.5 (0.9)	3.6 (3.6)
Having a positive/negative experience with the previous formative/summative assessment	3.5 (1.0)	–	3.7 (1.0)

A factor analysis was run on all items concerning reasons for completing OFAs. The scree plot showed a hitch on the fourth factor, and therefore we chose a factor solution with three factors, which explained 51% of the total variance and the three factors having eigenvalues of respectively 4.56, 3.96 and 2.98. For interpretation of the factors, we first determined for each item the highest factor loadings (boldface values in Table 4). Factor loadings smaller than 0.40 are presented in grey and cross loadings are presented in black.¹

Table 4.  Factor loadings of exploratory factor analyses on items concerning reasons for completing OFAs and the added value of OFAs with ML estimations and Oblimin rotation

	Factor
	1	2	3
I have completed the online formative assessment urinary system for …
Checking whether I studied sufficiently	0.09	0.46	0.48
Repeating and rehearsing the course material	0.13	0.52	0.27
Getting insight into the form and content of the summative assessment	0.22	0.44	0.10
Discovering gaps in my knowledge and skills	0.34	0.91	0.24
Understanding what elements of the course material are important	0.83	0.40	0.12
Having a positive/negative experience with the previous formative/summative Assessment	0.13	0.25	0.82
The online formative assessment urinary system had added value, because then I could …
Check whether I had studied sufficiently	0.23	0.29	0.45
Repeat and rehearse the course material	0.37	0.56	0.30
Get insight into the form and content of the summative assessment	0.56	0.24	0.33
Discover gaps in my knowledge and skills	0.58	0.62	0.19
Understand what elements of the course material are important	0.94	0.33	0.25
I will complete (at least one) online formative assessment, because …
Then I can check whether I had studied sufficiently	0.42	0.33	0.53
Then I can repeat and rehearse the course material	0.18	0.56	0.40
Then I can get insight into the form and content of the summative assessment	0.66	0.41	0.30
Then I can discover gaps in my knowledge and skills	0.52	0.66	0.38
Then I can understand what elements of the course material are important	0.88	0.32	0.26
I had a positive/negative experience with the previous formative/summative assessment	0.51	0.37	0.83

The items with the highest factor loadings on the first factor were the three items concerning understanding what elements of the course material are important and two items concerning getting insight into the form and content of the summative assessment. Based on these high loading items we interpreted the factor as using the OFAs to collect feed up information. The OFAs are used to see what the format of the assessment will be and what course material will possibly be addressed. In other words, students gain information about what the goals are and what is expected of them.

Second, the items with the highest factor loadings on factor 2 were the three items concerning repeating and rehearsing the course material, the three items about discovering gaps in knowledge and skills and surprisingly one item about getting insight into the form and content of the summative assessment. Based on the first six high loading items we interpreted the factor as using the OFAs to collect feed forward information. The OFAs are thus also used to learn the course material (move forward) and to know what elements of the course material would need to be studied more thoroughly. The fact that also one of the items of to collect feed up information loaded highest on this factor, in combination with a second item cross loading on this factor, is not that strange as feed up and feed forward are not independent elements as feed forward is about how to reach the goal more closely.

Third, the items with the highest factor loadings on the factor 3 were the three items concerning checking whether a student studied sufficiently and the two items concerning previous experiences with the formative or summative assessment. Based on these high loading items this factor is interpreted as students collecting feed back information. First, previous experiences already indicated how a student is going and are thus used in deciding whether using the OFA is needed, and second when the student completed the OFA, based on the outcome the student again checks where (s)he stands in relation to the goal. In total, 8 of the 17 items cross loaded on a second factor, indicating that these items were not strong in distinguishing the different factors.

Reasons for not completing the OFAs

Table 5 provides the medians of the different reasons for not completing the OFAs that were found in the qualitative measurement. From the table it can be seen that in general most reasons have reasonably low medians, indicating that not all reasons seem to play a role for students. Given the small N for the items of the second column of Table 5, only the items from the first column were used in the factor analyses. This yielded a factor solution with only one factor, indicating that no clear underlying factor structure could be found for these items.

Table 5.  Medians for reasons for not completing OFAs in terms of means and standard deviations (N = 29; N = 9)

Reasons	Not to complete the OFA kidney (retrospect)	Not to complete OFAs preparing for the final test (prospect)
Having too little time/making the OFA takes too much time	4	2
Having completed the test items below the micro lectures already	4	2
Already knowing the course material well enough	2	2
Finding the OFA insufficiently representative of the summative assessment	2	3
Experiencing technical problems	1	1
Preferring to learn from other sources	3	2

Discussion

This article explored student reasons for using OFAs when preparing for a summative assessment in a physiology course for biomedical sciences. Based on qualitative data gathered through open questions, a category system was developed that described the different motives students gave for making use of the OFAs, or not. In total six different reasons for making use of the OFAs were found, and seven reasons for not using them. Factor analyses on these seven reasons did not result in a clear factor structure for the function of not completing the OFAs. For the six reasons for completing the OFAs, three factors were found. Based on the work of Sadler (1989) and Hattie and Timperley (2007), the factors were interpreted as describing the underlying motives of collecting three types of information: (a) feed up, (b) feed back and (c) feed forward. Sadler (1989) argued that in order for students to use feedback for self-regulation in their learning process, they should seek information about (a) what the goals, standards and criteria are, (b) how their current performance relates to these criteria and (c) what they can do to close the gap between their current performance and the standards. Hattie and Timperley (2007) named these three elements feed up, feed back and feed forward. Our findings thus indicate the feedback function being an important explaining mechanism for the positive effects of OFAs on summative exam scores, rather than just increasing time on task (Cook et al. 2010) and preventing procrastination (Peat & Franklin 2002).

When we compare the three functions of OFAs to other studies investigating OFAs, several consistencies appear. These studies explored the student experience of OFA, but did not aim principally to gain insight into the specific functions. For instance, in line with the notion of feed up, Sly (1999) argued that an important function of formative assessments is that it gives students an idea about what is expected of them in the summative assessment in terms of both content and form. Also, Henley (2003), based on anecdotal evidence from informal student feedback, described two ways students could use the OFA. In the beginning, they used it after they studied the course material to check whether they understood the course material properly (i.e. feed back). Later in the course they used the OFA to guide their learning, i.e. before studying the course material (i.e. feed forward). Unfortunately, the design of our study does not allow us to draw such sequential conclusions as to how student use of OFAs changed during the course. Still, in line with Henley's (2003) finding, Walker et al. (2008) also found that students used e-assessments to identify areas of strengths and weaknesses in order to study further the course material (i.e. feed forward). Thus, the three factors we found can explain the preliminary research findings of other studies as well, suggesting that our findings are not specific to our sample only.

Concerning the findings of the present study about reasons for non-completion of the OFAs, it is important to bear in mind that some of the students indicated that they had already used the formative questions in another part of the electronic learning environment. In other words, these students did use the formative questions but not in the form of an OFA. Therefore, these findings might not be interpreted as if the students were uninterested in practising questions at all. That being said, our findings are now discussed in relation to findings on non-attendance in medical education, as this might give some indication of the extent to which our findings are specific to the use of OFAs or possibly can be generalised to students not being willing to participate in learning activities in general. For instance, Mattic et al. (2007) found that students reported both student-related and teaching-related factors for not attending lectures. In the present study “Finding the OFA insufficiently representative of the summative assessment,” could be considered a teaching-related factor and the other reasons (with the exception of technical problems) can be considered to be student-related, as these mainly describe students’ learning preferences. This is also in line with the findings of Billings-Gagliardi and Mazor (2007), who found that student decisions to attend lectures were among others based on personal learning preferences and learning needs at a particular time. Finally, “Experiencing technical problems” is neither a teaching-related nor student-related factor and is therefore interpreted as a mode-related factor. We thus conclude that students’ reasons for not using OFAs can be student-related and/or teaching-related, but can also be mode-related, which might be specific for OFAs.

Implications

In general, the findings of this study indicate that students used the OFAs for acquiring information with respect to understanding what is expected of them on the summative assessment, both in terms of form and content (feed up), and to what extent they have already acquired the course material (feed back), and to what extent they need to study further topics (feed back). In other words, they confirm the notion of Gibbs and Simpson (2004) that not only summative, but also formative assessment strongly drives learning. For the use of OFAs in practice, this indicates the importance of constructive alignment of formative and summative assessments as students use information from the formative assessment to self-regulate their learning activities in preparing the summative assessment, i.e. “we have first to be clear about what we want students to learn, and then teach and assess accordingly in an aligned system of instruction” (Biggs 1999, p. 8). It is thus important that formative assessments clearly reflect the learning objectives and therefore the content, level and types of questions of the summative assessment, in order to provide students with the opportunity to use properly the information that is acquired to prepare for the summative assessment.

With respect to the reasons for not completing the OFAs, we think some reasons might be overcome and some might not. As addressed above, the formative assessment not being representative of the summative assessment is a reason that might be overcome with strong constructive alignment. Also, the aim can be to avoid technical problems when using the OFAs. However, the student-related reasons indicate that OFA might not suit the preferred learning activities of all students. Therefore, it is important to design rich (online) learning environments so that students can study course material and prepare for assessments in a way that matches their preference.

Limitations

The design of this study has some drawbacks that should be kept in mind when drawing conclusions from these findings, based on which suggestions for future studies are provided. First, the present study was conducted during one course only and study strategies can differ considerably between courses (e.g. Kadri et al. 2011). Therefore, these reasons should be viewed as course-specific. Also, concerning inter-generalisability, future research is needed to validate the three functions that were found in order to see whether these motives also play a role in other courses with OFAs specifically, or other non-compulsory study activities in general.

Second, in the present study, only a small number of students had not completed the OFAs, so we had only minimal data about them. In future studies, it would be interesting to collect more data from these kinds of students so that for the reasons for not completing the OFAs general motives or functions underlying the separate items could also be investigated. Also, in the present study, students could use the questions from the OFAs in combination with viewing the micro lectures, which makes the group of students not completing the OFAs a rather heterogeneous group. For research purposes it would have been better if the OFAs were the only way to access the test questions. Finally, future studies might investigate whether different groups of students can be described based on their scores on the different motives using for instance cluster analyses. This would provide insight into how the three functions are related within students.

Conclusion

The results of this article revealed different student reasons for completing or not completing OFAs. More specifically, despite the fact that no clear factor structure was found for reasons for not completing, it is concluded that these reasons can be student-related, teaching-related and/or mode-related. As regards the reasons for completing three underlying types of information we found that by using the OFAs, students acquire: (a) feed up (what is expected of me in the summative assessment in terms of content and form?), (b) feed back (to what extent have I already mastered the course material?) and (c) feed forward (do I need to study more and if so, what do I need to study?). These findings can be related to findings of both Sadler (1989) and Hattie and Timperley (2007) and confirm preliminary indications about the use of OFAs (Sly 1999; Henley 2003; Walker et al. 2008) and findings about motives for lecture attendance (Billings-Gagliardi & Mazor 2007; Mattic et al. 2007). The results of the present study yield important theoretical insights into why students do or do not use OFA and indicate that, for practice, attention should be paid to constructive alignment of instruction, formative assessments and summative assessments (Biggs 1999; Gibbs & Simpson 2004).

Acknowledgements

The authors would like to thank Joop Hox and Urbano Lorenzo Seva for their valuable help and advice with respect to the data analyses.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the article.

Notes

Notes

¹As the data violated the assumption of normal distribution, we also ran an exploratory factor analyses for categorical data using the software package FACTOR. The results from these analyses showed an identical factor solution, i.e. all variables loaded highest on the same factor as in the original analyses. Therefore, we report the results of the original factor analyses.

Glossary

Self-directed learning: A process in which individuals take the initiative, with or without the help of others, in diagnosing their learning needs, formulating goals, identifying human and material resources for learning, choosing and implementing appropriate learning strategies, and evaluating learning outcomes.

Knowles M. Self-directed learning: A guide for learners and teachers. New York, NY: Associated Press; 1975 p 18.

Self regulation: Self-generated thoughts, feelings, and actions that are planned and cyclically adapted to the attainment of personal goals.

Zimmerman BJ. 2000. Attaining self-regulation: A social-cognitive perspective. In: Boekaerts M, Pintrich P, Zeidner M, editors. Handbook of self-regulation. Orlando, FL: Academic Press. pp 13–39.