The role of the assessor in the assessment of practice: An alternative view

Abstract

Background: Traditionally, the assessment of the competent practical performance of student health professionals places a high level of confidence upon the ability of a second party to judge whether students are able to apply theory in practical contexts. More often than not, the specific assessment tool has a clear focus on the conduct of skills to explicit performance criteria. However, what has proven particularly problematic is the ability to devise a tool to assess the less overtly procedural qualities, such as professionalism, interpersonal skills, clinical reasoning and so forth. When such assessment tools are designed there is also particular challenge to counter the potential for conscious or unconscious bias on the part of the assessor.

Aim: With direct reference to enactivist theory, and using illustrations from undergraduate medical, nursing and physiotherapy education, it is argued that in the conduct of any assessment the assessor is as much an integral part of the assessment as is the student.

Conclusion: The enactivist perspective has clear implications for the role of the assessor in the preparation, conduct and outcome of the assessment of practice-related activities.

Theory, practice and assessment: The orthodox position

The orthodox position, in respect of the relationship between theory and practice, has three key tenets. The first of which is that any theory that underpins an occupation and the practice of that occupation are distinct entities separated by a gap. The second is that any difference between theory and practice, as represented by that gap, can be identified and isolated. The third is that the size of the gap can correspondingly be increased or lessened.

Table

Practice points

The components of an assessment are inseparable. The assessor does not stand outside of the experience that comprises assessment. The influence of the assessor is most acute when less tangible and aspects of practice are the focus. Enactivist theory contributes to the understanding of the challenges presented by attempts at assessing practical performance.

These three entities, theory, practice and a gap constitute the basic components of the orthodox position. They are components that instil a tangible quality to the relationship between thought and action and suggest that somehow the relationship is amenable to manipulation.

The orthodox position relies heavily upon reductionism, which is by definition divisive, thus the orthodox position both creates and then reinforces a false dichotomy between theory and practice (Hoare & Nowell Smith 1971; Moccia 1992; Carr 1995).

With direct reference to student assessment, and as if to emphasise that dichotomy, different methods are specifically used for the assessment of theory or alternatively, practice. Traditionally, pen and paper testing is employed to assess propositional knowledge and observation by another person of actual or simulated demonstration on the part of the student is commonly used to assess practice.

The seamless and interwoven experience that comprises assessment

The literature that pertains to the education of health professionals regularly expresses concern about equity in the assessment of practical skills and a subset of that concern is the challenge to minimise bias on the part of an assessor.

For example, the tried and tested assessment strategy known as objective structured clinical examination (OSCE) has become a central part of the assessment of medical students (Davis 2003). The OSCE is also gaining popularity amongst other health professions in their search for an impartial method of assessing a range of clinical examinations to predetermined criteria. However, whilst the OSCE is considered to be high in terms of objectification, it is also fully acknowledged that despite their best efforts, it is not possible for an assessor to behave objectively (Wilkinson et al. 2003). It was also noted in the context of medical education that even highly experienced assessors who work from explicit criteria reach different end-point decisions of the same assessment (Perkins et al. 2001).

Similarly, in the context of physiotherapy education Lewis et al. (2008) evaluated the interrater reliability of a standardised assessment tool amongst different assessors. They found that the interrater reliability amongst assessors was poor and concluded that one reason for this might be that: ‘ … individual clinical educators had differing opinions as to the patient's main problems and the type of treatment interventions that were appropriate’ (p. 127).

In relation to the assessment of qualities, such as professionalism or effective interpersonal skills, Kuper et al. (2007) investigated the role that assessor perception played in attempts to evaluate the identification and application of the interpersonal quality of empathy by medical students in patient contact. The authors concluded that: ‘These perceptions may differ radically not only between the trainee and the patient, but between either or both of them and an examiner who is observing the encounter’ (Kuper et al. 2007, p. 1122).

In nursing education, a systematic review of the literature was undertaken in order to establish if clinical competence could be assessed. They found that there was widespread support for the requirement that nurses be clinically competent. However, they also found that there was a: ‘ … lack of consensus about the definition of clinical competence and the means by which it should be measured, the likelihood that vital concepts are excluded and irrelevant concepts are included in existing clinical competence assessment instruments is very high’ (Watson et al. 2002, p 423).

The role of the assessor in the orthodox position

An allegedly important part of the assessment of practice is to not only determine if a student can do something, but also to determine if that student is able to make effective links between the theory and the practice of their particular occupation.

In the orthodox position, assessment techniques that concern practical activities more often than not are those that place a high degree of confidence in the testimony of another person, usually based upon the observation of performance. These techniques demand a demonstration by the student to explicit performance criteria and require the student to provide either, an oral or written explanation or both.

However, within the framework of orthodoxy it can be argued that demonstration or exposition should not be taken as confirmation of agreement between thought and action, and that the ability to demonstrate competence does not always fit neatly with the articulation of theoretical explanations (Wittgenstein 1953; Sandelands 1991).

In the orthodox position, as a valid form of confirmation of synergy between thought and action, reliance upon personal explanations is considered to be flawed. This is because the ability to demonstrate competence, to perform, does not fit neatly with the articulation of theoretical explanations and explanations cannot be regarded as evidence of the ability to perform.

For example, to determine if a medical student is able to provide effective pain relief for a patient, merely asking a series of questions about a hypothetical patient does not provide adequate evidence of the ability of that student to apply theory to practice. If the student is then asked to explain how pain relief works, this gives only insight into their ability to recount the theory and the application of that theory to the hypothetical patient.

If we were then to ask for a demonstration of how to provide pain relief to an actual patient, is competence judged by the process of attempting to provide pain relief, or by the outcome of the intervention? If the patient continues to experience pain, what does that convey about the general ability of that student to provide pain relief?

The problem has little to do with the theory–practice relationship. Rather, it is a relationship between formal knowledge and action. This is a commentary not on the appropriateness or not of the theory but of the ability of an individual to assimilate and use that theory in the practical context. It is apparent that in the orthodox position, the high level of confidence afforded to attestation by another person, is a confidence that could be misplaced.

In the assessment of physiotherapy student's practical skills, Clouder and Toms (2008) acknowledged the inherent problem of relying upon solely observation or explanation as indicators of competence. In their study, they evaluated the contribution that both the ability to observe and then question students made to the depth and accuracy of assessment decisions. They reported in the feedback from assessors [Clinical educators] that: ‘Clinical educators perceived that together the two components were assessing different attributes, although there was a ‘need for interaction between both’ (p. 32).

It is evident that these authors realised that the orthodox position presented clear challenges in their attempt to confidently assess distinctly procedural activities. However, when the less tangible aspects of practice, such as professionalism, clinical judgement or interpersonal qualities are the focus of the assessment, the ability to make an accurate assessment becomes even more problematic.

When the perspectives in the previous two sections are combined, it is clear that there is a real problem in trying to disentangle the various components, including the role of the assessor, from the total experience in the assessment of practice.

This suggests that attempting to understand the assessment of practice from the orthodox position will inevitably prove to be incomplete. It is proposed that to understand the seamless and interwoven nature of the total experience that comprises assessment, the theory of learning known as enactivism will offer an alternative and complementary perspective.

Enactivism: A personal interpretation

Enactivism is a theory which openly incorporates ideas from other disciplines, such as psychology, biology, ecology, philosophy and anthropology. The eclectic nature of enactivism means that elements of enactivism can also be found in embodiment and experientialism (Reid 2001). In education, the notions contained within enactivism are an extension of the constructivist perspective and build upon the dynamic nature of constructivism, whilst also explaining non-cognitive knowing, intuition and the role of emotions in learning.

Enactivism acknowledges that the learner and the external world are part of the same ecological system in which the learner, as a living organism, has the capacity of autopoiesis; the ability to continuously change whilst still maintaining wholeness as a self-organising system. Moreover, in order to survive, a self-organising system must have the structural capacity to respond to change or effect change in the medium, or environment, in which the organism exists. In the context of education, this capacity for adaptation is best understood as learning (Maturana & Varela 1992).

Enactivist theory advises that when we perform an action, we do not stand outside of that action and passively observe it. We are intimately part of the action, whilst at the same time influencing the progress of the action. Simultaneously, our perceptions of our action are guided by the action itself, which in turn influences our understanding of that action. In the performance of practical activities, cognition, emotion and sensori-motor activity guide each other in a simultaneously and integrated circularity.

The circularity between thought and action, central to an enactivist explanation of learning cements an individual with the external world as an integral and evolved component of that world (Varela et al. 1997). In the context of vocational education, enactivism places a much stronger emphasis upon learning as a holistic experience. For enactivism, learning is a process that involves the teacher, the student and subject boundaries merging in a constant process of change (Maturana & Varela 1992). Thus for enactivism the total experience that constitutes learning, there can be no arbitrary separation into components.

The role of the assessor in the enactivist position

In enactivist theory, any relationship between thought and behaviour is conceptual and not logical. Further, the reasons for action are considered an integral part of that action and not a precursor to the action (Shotter 1975). Therefore, any link between thought and action is contained in the meaning that they produce for the individual and not in formalised links that may be explained to, or observed by another person (the assessor). Enactivism also suggests that the ability to deliberate before doing is a part of being autonomous and the reasons for actions are not always something that is thought about in advance of that action, nor spoken about when that action has been completed. In respect of the role of a second party, Maturana observed that: ‘ … the listener, the questioner, decides what an explanation will be’ (Maturana 1987, p. 67).

Thus, any assessment of the relationship between theory and practice becomes the perceived difference between what the observer wishes or expects to see in the form of success criteria. What the student does may, and only may, consciously or unconsciously satisfy those criteria. Thus, enactivism eschews the reliability of observation in that there is clearly more to individual experience than can ever be observed, and for the observer the change sometimes is not entirely discernible.

Assessment and enactivist theory: Some implications

The first implication is that assessors should regard assessment as partnership with the student. The primary educational purpose of any practical placement in a clinical setting is the opportunity for students to learn the skills, knowledge and attitudes pertinent to their future professional role. As part of that role, the student must develop the ability to understand the relationship between theory and practice, and that in turn requires that students make links between different notions and concepts. This integration on the part of the student is a form of understanding that requires learning at a deep level (Hillier 2002).

A further implication is that students should not only be passive partners in the assessment process but they should be a highly active partner in the same way that they are active in their own learning. It is evident that in practical settings, students intentionally make their own choice of what will be learned in that: ‘He [the learner] need not … learn something he is not ready for, thinks uninteresting, frightening or unnecessary. The learner makes his own curriculum’ (Becker 1972, p. 99).

A third implication is that if students are to be assessed, they should have a role in determining which will be assessed, the frequency by which they will be assessed, and the means by which they should be assessed. Mindful of the mandatory requirements of formal assessment, the individual student should be able to be more involved in the selection of a range of assessment of methods and determine the timing of an assessment.

One of the factors known to detract from deep learning and encourage superficial or strategic learning is regular and frequent summative assessment (Marton & Saljo 1997). Conversely, formative assessment, as long as it is accompanied by feedback, encourages deeper learning on the part of students (Rushton 2005). Thus, a fourth implication for the design of vocational programmes is that a move towards an assessment culture, one that encourages learning should replace a testing culture in the context of practicum assessment. Further, for at least one clinical experience, the student should be liberated from the requirements of an imposed summative assessment.

A fifth and final implication of enactivism is that it must be openly accepted that the components of assessment are interwoven and inseparable. Therefore, in the context of professional education, attempts to adopt overtly reductionist approaches to the assessment of procedural skills, perhaps exemplified by the somewhat mechanistic standard-based approach found in many forms of vocational education, should be reconsidered.

Conclusion

In the orthodox position, the assessment of practice-based learning places a high level of confidence on the close relationship between explanation, understanding and performance. Paradoxically, however, this position also maintains that the ability to demonstrate or explain should not be construed as confirmation of synergy between thought and action or of the ability of the individual to be able to link theory with practice. Correspondingly, the ability to perform competently does not always mean that a person will be able to articulate the relevant theoretical explanations (Wittgenstein 1953; Sandelands 1991).

It may be observed that in practical summative forms of assessment, the student may only perform to the level that they needed to perform to pass the particular assessment. What happens before and after the assessment might or might not be different, but it would almost certainly be a different type of performance (Biggs 2003). Therefore, for the assessor, the assessment of practice is a highly subjective act. The assessment of practice is an act of observation for which the observer is as much part of the activity as the observed, and the performance which is being observed (Maturana 1987).

In the context of performing a social action and this includes skilled performance in the conduct of an occupational role, any theoretical knowledge that is acquired is not necessarily evidenced by external and observable actions. Formal knowledge is not demonstrated in practice only that component which an observer may or may not take to be evidence of understanding for only the learner can engage in the experience of learning, and only the learner can sense a link between theory and practice. It is the learner, as the doer, who is the final arbiter of perceived link between theory and practice (Habermas 1974).

In the conduct of an assessment, whether theoretical or practical, any belief that the assessor is somehow unconnected or even that the assessor could ever be unconnected from that which is being assessed; is based on a false premise. Furthermore, in the attempt to assess the quality of less tangible qualities, such as professionalism or interpersonal interactions, medical educators should abandon reductionist approaches to assessment as exemplified by a standard-based approach to the assessment of competence. Finally, because of the interwoven relationship between the assessor, the assessed and that which is being assessed educators should not be unduly self-critical with regard to failed attempts at objectivity.

More importantly for the professional education of groups, such as health professionals, lawyers, nurses, teachers, amongst others, reductionist approaches to the assessment of key global qualities, such as professionalism, interpersonal skills and so forth, methods that openly acknowledge the wholeness that comprises skilled, competent and professional performance should be employed (e.g. Polyani 1967; Schön 1987; Eraut 1994; Beckett 2004; Benner 2004).

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the article.