Use of certainty-based marking in a second-year medical student cohort: a pilot study

Abstract

Background

Assessments which consider both competence and confidence attempt to provide insight into actual performance in order to optimize physician capabilities, providing motivation and direction for future learning. The aim of this project was to assess medical students’ thoughts and opinions of the utility of a certainty-based marking (CBM) protocol with respect to improving their learning experiences.

Methods

Second-year medical students at the University of Queensland were provided with a series of optional online formative assessment tools, in the form of 10 sample questions, to support their current module learning outcomes. During four consecutive weeks, CBM was offered on weeks 1, 2, and 4, with week 3 being provided in the usual question-answer format. A mixed-method survey was distributed at the conclusion of the trial period to obtain feedback on the students’ impressions of learning via this technique.

Results

Of the 400 students, 15%, 11%, 9%, and 8% used the resource over the four-week period, respectively. During the four-week module directly prior to the test module, 46%, 44%, 44%, and 40% of the students accessed the sample questions which were delivered in the usual multiple choice format. A majority of the students either agreed or strongly agreed that CBM was easy to understand (52%) and useful (57%), but took more time (67%) because they needed to consider their certainty level for every question (76%). A number of students (43%) also stated that CBM affected their attitudes toward decision-making, while 86% thought it would be most useful for revision as opposed to an examination format.

Discussion

Despite the inherent benefits of gaining experience in higher order thinking processes, students were less likely to participate in the CBM tasks than standard multiple choice, even though these did not count toward their final grades.

Conclusion

Utilizing such practices at the beginning of an educational program may minimize apparent resistance and alter learning practices to become conducive to deeper levels of learning. This has been corroborated in other studies aiming to encourage similar higher order cognitive processes.

Keywords:

• assessment
• medical education
• certainty-based marking

Introduction

Certainty-based marking (CBM), formerly known as confidence-based marking, is a tool used to encourage students to think more carefully prior to deciding on an adequate response during assessments. It encourages reflection on reasoning prior to making clinical-based decisions, because the answers also require an indicator of level of confidence in their response. The process is thought to enhance deeper levels of learning at the expense of commonly utilized rote learning practices. These apparently superficial techniques create a lack of thorough understanding of issues, despite being effective in achieving satisfactory grades when assessed by simple yes and no responses.¹

According to Bloom’s Taxonomy of Learning, knowledge is the simplest level of learning to acquire. Here information is recalled exactly as it is presented and requires memory of definitions, formulas, or procedures. These levels progress through comprehension, where students state relevant material in their own words, and application where rules are applied to a problem without being explicitly stated. Analysis requires yet deeper levels of learning where relationships are drawn and complexity is broken down. Evaluation exists at the top of this hierarchy, where judgments are conceived based on the worth of something for a specific purpose.² Having students evaluate their progress in the form of CBM demonstrates higher order thought processing than standardized direct question and answer formats.¹

CBM in different forms is decades old. Generally speaking, students are requested to grade their confidence in the answer they have stated at one of three levels, ie, 1, 2, or 3. If correct, this becomes the number of marks they receive. However, if they are incorrect, at the lowest level there is no penalty, followed by -2 and -6, respectively. According to game theory, level 2 should be chosen when students are over 67% certain and level 3 if greater than 80%.³

CBM ensures that students differentiate responses as to whether they are based on sound knowledge or may be at risk of inaccuracy. It encompasses the importance of vigilant thinking and reassurance, which assist to link more diverse aspects of information. In this process, confident responses receive higher scores if correct, but incur higher penalties if incorrect. A qualitative evaluation study by Issroff and Gardner-Medwin⁴ documented quotes by students such as “it stops you making rush answers”, “you can assess how well you really understand a topic”, and “it makes one think … you are forced to concentrate”.

A key feature of these methods is their ability to motivate students via awarding extra marks for certainty of a correct response, while penalizing an incorrect but certain response. When choosing a marking scheme, it is advisable to focus on the way confidence is elicited to ensure proper motivation.⁵ Negative marking schemes have been acknowledged to discourage guessing, but this is not necessarily the case unless there is a positive motivation to perform otherwise.⁶

Subsequent documented issues have arisen in regard to using CBM. There is concern that overly self-confident or less confident students may be disadvantaged by this style of assessment. However, it is argued that these personality styles should become more self-aware through the process, if they are not to be penalized. Essentially, students who recognize their rationale or lack thereof benefit, while those who are consistently confident or apprehensive do not.¹

Results from CBM assessments may also be utilized effectively for evaluations of course material, where questions eliciting high levels of incorrect confident answers can highlight common misconceptions to educators.⁷ The process of self-explanation, which grows from the need to make confident judgments, can assist the student to refine their methods of problem-solving.⁸ These constructed activities, directed at recognizing missing or unreliable information, can lead to modification of existing knowledge and creation of a new understanding.⁹

The use of CBM in health care training has important implications, particularly in clinical situations, where any risk of error should be recognized and acted on accordingly. It is likely to be far more detrimental to possess a confident belief in an erroneous judgment than the ability to recognize a lack of knowledge in that area.¹⁰ Hence, the aim of this study was to assess students’ opinions of CBM as a useful adjunct in enhancing current curricula, as well as to explore the effectiveness of its implementation.

Materials and methods

Second-year medical students from The University of Queensland were asked to volunteer to make use of CBM format multiple choice questions to assist with formative evaluations of their own levels of knowledge. Course content is generally presented as sets of modules throughout the first two years of the MBBS program. Each week, a series of 10 multiple choice questions is made available via the Moodle on-line learning system to support student education. The Moodle 1.9 platform features a specific quiz module which incorporates CBM. Detailed information on the use of this media is available.¹¹

The current project converted three of the four weeks of formative assessment quizzes, within a nervous system module, into CBM-style criteria, namely, weeks 1, 2, and 4. A mixed-method design questionnaire was also available for the students to provide feedback on their experience and opinions of CBM, if they so wished. Follow-up survey questions included whether CBM was “easy to understand” or “useful”, and whether it “affected attitudes” or “length of time to provide an answer”. Students were also asked if CBM “increased their consideration of certainty” or if they felt it to be “a waste of time”.

Student participation was entirely voluntary and all relevant data collected were deidentified. Prior to their involvement, participants were provided with a detailed information sheet outlining the project and also highlighting the many potential benefits of considering certainty when problem-solving. These included but were not limited to increasing reflective practice and encouraging the development of higher-order cognitive processing,¹² subsequently enhancing clinical competence and reasoning skills.¹³ Ethical approval was obtained from the University of Queensland’s ethics committee. Consent to utilize the deidentified research data arising from this project was obtained from the students prior to their participation.

Results

Of the 400 students, 15%, 11%, 9%, and 8% used the resource over the four-week period, respectively. During the four-week module directly prior to the test module, 46%, 44%, 44% and 40% of the students accessed the standard multiple choice method sample questions. Only about half of the participants elected to complete the survey (n = 39). The results are outlined in Table 1.

Table 1 Student responses relating to CBM

Responses on additional time required to complete CBM quizzes as opposed to standard formats ranged from 30 seconds up to 30 minutes, with the most common response being around 50% of additional time required. Opinions of the usefulness of CBM were mixed with a variety of responses. Positive comments included “it assisted in interpretation discrepancy”, “a good tool to revise learning”, and “seemed to make me more analytical with every question and work through the information more precisely”. Negative responses consisted of “most ridiculous thing ever” and “would be too time-consuming during exams”, while mixed reactions comprised thoughts such as “a lot more work than the standard quizzes, but more informative” and “best for clinicals but not exams”.

Discussion

The ability of physicians to recognize when they are not certain of an adequate treatment protocol is vital, especially in an emergency situation. A student’s confidence about incorrect responses may be far more unfavorable than recognition of uncertainty with a particular answer.⁵ Utility of CBM during medical education attempts to facilitate this awareness process. As such, this project aimed to explore the implementation and uptake of such an approach, as well as the students’ impressions of its effectiveness in improving their learning outcomes.

Traditionally, CBM has been made use of in true/false questions, although no particular issues have been documented by its use with numerical, best-of-5, or extended matching questions.¹⁴ Typically traditional multiple choice types of assessment may lend to situations where a student is able to answer a question, once options have been presented, due to a latent recognition process. This form of cueing can be particularly problematic when diagnostic reasoning is being assessed due to premature closure prior to the correct diagnosis being considered, which is a common reason for error in clinical practice.^13,15 Use of CBM may minimize this effect because more careful consideration is required once certainty-based parameters are included. The majority of our participants (76%) indicated that they needed to consider the certainty level for every question. Interestingly, a small portion (29%) deemed consideration of certainty to be a waste of time.

Extra time required to complete an assessment is another matter to be considered when comparing CBM with conventional testing. The majority of our cohort (67%) specified more time was required to ensure an accurate response. An evaluation study conducted by Issroff and Gardner-Medwin documented that many students stated they sometimes changed their responses when having to consider certainty around their answers.⁴ Considering the large portion of our participants indicating their preference for CBM as a revision tool as opposed to being utilized in examinations, the issue of additional time required to respond to questions may have impacted overall impressions and participation. Despite inherent concerns on time consumption, another study assessed the reliability analysis of examinations and concluded that less than a third of the number of questions was needed for equally reliable assessment data using CBM techniques.¹²

Extraordinary time constraints are often in place in medical practice, where processing and retrieving information occurs continually at the time of action.¹⁶ It has also been established that time constraint is one of the major factors limiting a physician’s ability to comply with preventative recommendations to their patients in practice.¹⁷ These internal time constraints may lead to damaging results in which patients and providers may both become victims.¹⁶ Should we be pushing these boundaries already at the commencement of the medical education process, selecting the specific value of tools in the interest of time?

Limitations of this study include potential selection bias, because only a small sample of the total student body made use of the CBM resource. Despite being informed of the potential benefits of CBM prior to its implementation, participation may have been improved if it had been provided at the beginning of the first year as opposed to the middle of the second year when students are more likely to be set in their thought processing.

Conclusion

Regardless of the inherent benefits of such an approach, students were less likely to participate even though the tasks were formative assessments which did not count toward their final grades. Perhaps using such practices at the commencement of an educational program would alter learning practices conducive to deeper levels of learning. This may also minimize resistance to the additional thought processing required in ascertaining certainty of core knowledge. In the words of the famous philosopher, Confucius, “When you know a thing, to hold that you know it. And when you do not know a thing, to allow that you do not know it. This is knowledge”.¹⁸

Acknowledgment

Production of this manuscript was supported by a Scholarship of Teaching and Learning Grant from the University of Queensland.

Disclosure

The authors report no conflicts of interest in this work.