The Role of Assessment in the Development of Judgement

Abstract

This paper introduces an integrated assessment model developed within a project management discipline stream in a Construction Management course. Following Boud and Falchikov (2007) this model starts with practice, that is, the actual ‘doing’ of project management as the basis for shaping assessment that equips students to learn for the rest of their lives. Practice is understood as a holistic conception of what professionals do in particular contexts, and a theoretical construct that provides a method of framing ways in which we can investigate the world (Schatzki, 2001). This approach opens the way for considerations of assessment that engage with, and cultivate, certain kinds of professional learning and identity formation including the development of judgement. Integral to the model is the non-sequential nature of assessment activities, evolving team formation and ongoing self and communal reflection. The paper concludes that the use of an authentic and integrated assessment model creates a compelling learning environment that contributes meaningfully to the development of skills, knowledge and identities for future professional learning.

Keywords:

• Project Management
• Professional Practice
• Assessment
• Judgement

Introduction

Project management is complex work. Shared with many professionals is a need for technical expertise and a body of knowledge, but that in and of its self is insufficient. Like all professionals, ‘professional practice requires a much richer set of phenomena ߞ a capacity to make judgements, sensitivity to intuition and an awareness of the purposes of the actions are all involved’ (Beckett and Hager, 2002, p.12). The knowledge and skill set envisaged by Beckett and Hager raises some particular challenges for those charged with the initial development of project managers in undergraduate degree programmes. As Boud and Falchikov (2006, p.402) argue:

Preparing students for lifelong learning necessarily involves preparing them for the task of making complex decisions about their own work and that of others and for making decisions in the uncertain and unpredictable circumstances in which they find themselves in the future.

Within higher education we have seen an increase in the scholarly attention given to the role that assessment can play to promote learning, including most recently the role that assessment can play in the development and practice of making judgements (Boud and Falchikov, 2007; Joughin, 2008). There is much evidence to suggest that advancements in assessments are taking place, including for example the incorporation of a greater variety of assessment modes, and a greater awareness of the need for formative assessment and feedback. This represents a significant shift away from traditional models of assessment where assessment practices have not attended to the role of student judgement, but rather as a means for students to attend to the judgements of others (Boud and Falchikov, 2007). The traditional assessment practices of examinations, tests and assignments often say much more about the disciplinary cultures and conventions than a manner through which to equip students for a life beyond the university context. As Candy et al. (cited in Boud and Falchikov, 2006, p.403) remind us, ‘…in real world contexts, they (students) must be able to judge or evaluate the adequacy, completeness or appropriateness of their own learning’. For the development of ‘professional’ project managers, cultivating and enabling the making of judgements is fundamental.

The assessment model, the focus of this paper, starts with professional practice, that is the actual doing of project management work as the basis for choosing and adapting assessment. We adopt Boud’s conception of practice as a pragmatic expression: ‘…representing what students and graduates do when they exercise their knowledge, skills and dispositions with respect to problems and issues in the world’ (cited in Joughin, 2008, p.30). For a professionally oriented course such as construction management, designing assessment that cultivates certain kinds of professional learning and identity formation has the potential to create a compelling learning environment that equips students to learn for the rest of their lives. Central to the model is the development of judgement, practised through non-sequential assessment activities; team work and in self and communal reflection.

The rest of the paper is structured as follows: in the first section we set the context of the project management units within a Construction Management degree course; we then consider what we mean by ‘practice’ and explore the implications for the development of assessment, where informing judgement is paramount. In the second section we outline the model of assessment currently used in a project management discipline stream and explore the key elements. We conclude with a brief discussion of some of the challenges of this model for students and teaching staff, and propose a research agenda as a way forward.

Context

Figure 1 Learning Framework ߞ Project Management (PM) stream

The project management stream of the Construction Management course at Deakin University consists of three units aimed at equipping future construction managers with a set of competencies defined by the nine areas of Project Management Body of Knowledge (PMBOK). The nine areas include: project human resources, communication, quality, procurement, time, cost, risk, scope and integration management.

In these units students are introduced to and have real opportunities to practise project management in a learning environment that replicates authentic, real life situations. All tasks are designed so that those undertaken in the first year provide the foundation for the subsequent tasks, and all tasks are interrelated across the degree programme. The two major tasks in the first unit are to develop a project team recruitment and development strategy, and project communication model based on interviews and collaborations with managers in construction industries. In the second unit students use the strategy and the communication model and apply them in the development of an operational plan for an educational project. They then conduct a quality audit of the performance based on previously planned projects. Two other projects in the second unit relate to residential structures for which students develop an environmentally friendly procurement strategy as well as time and cost management plans. In the third unit students’ work is based on a commercial scale project for which they develop a project brief and project risk management plan for its construction phase. The final project in the project management discipline stream integrates all the project management processes and all PMBOK areas of knowledge. The project is undertaken entirely in a virtual space, where students use contemporary information and communication technology such as the social software Elluminate Live, to collaborate and share the project’s outcomes.

A Practice-based Approach

Underpinning the approach to assessment developed in this paper is a view of professional practice, that is, what construction/project managers actually do in the everyday world of construction management. Practice theory is helpful as it provides a useful framework through which to examine project or construction managers’ engagement within their setting, specifically the emergent capacity of decisions or practical judgements. Of particular relevance to the model of assessment, practice theory enables us to better understand the development of knowledge in social and physical environments as they occur.

While it is not within the scope of this paper to do justice to the rich philosophical and sociological accounts of practice, the concept of practice as developed here requires some elaboration. In the introduction we drew on Boud’s (cited in Joughin, 2008) pragmatic account of practice as a useful starting point. Beckett and Hager (2002) deepen that initial take on practice. They argue:

It is certainly not merely ‘technique’, although technical expertise (certain sorts of skilful dexterity, involving manipulation of materials, objects, processes and ideas) is essential. Technique is a necessary but insufficient component of practice. Practice involves a richer set of phenomena: a body of knowledge, a capacity to make judgements, sensitivity to intuition, and an awareness of the purposes of the actions are all involved in some way.

(Becket and Hager, 2002, p.12)

Practice as understood as a complex phenomena is shared by Schwandt (2004) who suggests that practice is closer to the Greek term praxis, in that: ‘praxis demands a particular kind of engaged, embodied and enacted judgement that links knowledge, virtue and reason. (p. 321). While Schön (1995, p.29) expresses it thus:

…our knowing is ordinarily tacit, implicit in our patterns of action and in our feel for the stuff with which we are dealing. It seems right to say that our knowledge is in our action. And similarly, the workaday life of the professional practitioner reveals, in its recognition, judgements, and skills, a pattern of tacit knowing-in-action.

While considerable variation exists, all emphasise practice as some kind of purposeful engagement with the world where practice and knowledge are inextricably linked and always situated, that is they have a specific ‘where’ and ‘when’. What is important here is the context-relativity of practical knowledge; that is, knowing is bound up with workplace settings or activity systems and particular norms and communities of practice.

In addition they each share a view of practice as necessarily embodied; practice involves the whole person including motives, feelings and intentions. This gives rise to professional identity formation as an important consideration in the development of professional practice. Beckett and Hager (2002, p.37) provide a useful account of how the exploration of professional identities, including the development of judgement, enables us to develop a more sophisticated concept of professional practice.

Other practice theorists such as Gherardi (2006) and Nicolini et al. (2003) share the centrality of an individual’s engagement within their settings. Like Beckett and Hager (2002), and Schwandt (2004), they argue that we cannot be understood as apart from our social and physical settings, but rather as embedded within them. Implicit within practice theory is a different conception of what it means to learn, and offers new insights into how students may become engaged within their environments, and how best to facilitate learning.

Recent work by Boud and Falchikov (2007) has been at the fore of reframing assessment from a practice-based approach that better equips students for a lifetime of learning and the assessment challenges that they will face in the future. At a keynote entitled ‘How can practice reshape assessment?’ given at the 2007 Australian Technology Network Assessment Conference, Queensland University of Technology, Boud made the following four observations that we find generative, and develop further here. First, a practice-based approach is anchored in the professional world, not the world of educational institutions. As such there is an authenticity afforded to the context which is often devoid in more traditional university contexts. Authentic practice exposes what Nicolini et al. (2003, p.22) refer to as ‘breakdowns and disturbances’; inconsistencies, paradoxes and tensions that are all fundamental and inescapable aspects of practice. This is redolent of Schön’s (1987) work on reflection-in-action as they provide powerful observational triggers for students to engage in reflexive learning. Schön’s work on reflection-in-action was a deliberate attempt to evoke learning from experience. Others such as Beckett and Hager (2002) have shown how such reflection enables us to transform experience into knowledge, which can then be represented and generalised to new contexts.

Second, practice focuses attention on work, which extends beyond the artefacts of the course and assessment criteria. When learning is anchored in practice, the frame of reference is quite different; assessment becomes more about developing habits, ways of working and thinking that serve internal objectives but are fundamentally useful in their future professional contexts.

Third, those actions have consequences beyond those of formal assessment requirements. For many assessment tasks, the consequence for students is a numerical outcome. In a practice view assessments provide an opportunity for professional and personal growth; it is no longer simply about a grade, but a more holistic account of students’ engagement in process, approach or behaviour that is generative for future professional contexts.

Finally, that judgement of those involved in a practice situation makes a difference to those involved. In assessments that mirror practice, judgements have an impact ߞ colleagues, clients and so forth, and enable students to develop a more sophisticated conception of professional practice.

In the section that follows we show how the above considerations reshape the model of assessment.

Model

The theoretical considerations for assessment raised by the adoption of a practice-based approach have implications for the nature of assessment activities developed. In this section we attempt to elucidate how such theoretical considerations have reshaped the model of assessment.

Figure 2 Project Management Assessment Model

The model represented in Figure 2 illustrates the learning phases for one assessment cycle for one project only. The number of project cycles is dependent on the year level but normally students would undertake two to four projects simultaneously. The assessment cycle requires student engagement in a collaborative learning process, whether in real or virtual space, and in substantive learning support measures such as continual feedback and consultations, as co-created by the teaching staff and students. In bringing together a practice approach and assessment, we draw attention to five key characteristics:

1. Assessment activities which are grounded in authentic project management work

Underpinning the design of assessment activities is a whole approach to curriculum designed to emulate the world of project management, that is, to create contexts, relationships and activities that get close to the kind of undertakings that are routine to project management work: lectures become project updates and briefings, tutorials become project meetings where projects’ specific tasks are worked out and final project outcomes are shared. The learning objectives reflect a cross-section of personal and professional objectives, project objectives and deliverables. While students are exposed to the relevant body of knowledge, it is partially stimulated by students who become aware of a need to know a specific theory or body of knowledge that might benefit their project. Typically the management of teaching and students’ learning is directed by a teacher, however, in this model the management of the learning environment is shared with students who may be managing multiple projects simultaneously. Consequently the ownership of learning shifts to students, where the learning, skills and knowledge application is imposed by real life professional dependency and situation.

In this model, traditional assignments become complex multilayered project products. Each project, for example, consists of a number of products: a detailed written component such as a project plan, a project briefing where the ideas and concepts are proposed, feedback generated by staff and other students, and a ranking of project products. Alongside the project products, students capture their own critical reflections around the kinds of professional judgement made, and the reflections of their community in a professional portfolio (journal).

2. Characterised by a non-sequential assessment activities

Traditionally, assessment tasks tend to be undertaken sequentially, where students complete one assessment before commencing on the next. In some units the assignment might be partially connected, so that the second builds upon the first, and so on, but more often the tasks are disconnected. The nature of development advocated in this article requires a substantial rethink of the usual linear deployment of assessment tasks. Specifically the development of judgement requires a model of assessment where ‘capability does not build linearly but through cycling through different tasks and returning to previous tasks when confronted with new domains of learning’ (Boud and Falchikov, 2007, p.186).

In this model, students work concurrently on a number of individual and group projects. The scope, objectives and deliverables are negotiated with students at the beginning of the trimester but may be renegotiated as the study progresses. Any progressive outcome produced individually or by one of the project teams has immediate application to one of the projects. The two individual and five collaborative projects are distinctly different, however there is a high degree of interdependence and integration, which allows some outcomes of one project to be used in another one, or at least they are recyclable. For example Project 2 might be on a performance review of activities planned in Project 1; Project 4 is a scaled up version of Project 3, but differs in that it focuses on specific project management aspects such as project procurement and project quality management; an individual project, Project 6 which aims to develop and maintain a professional portfolio is linked with a collaborative project, Project 5, on planning and delivery of a seminar for construction management professionals in the role of portfolios in professional development.

3. Involves students in multiple team formation

Traditionally students form teams in order to complete one designated team assessment task during a trimester; typically, students would work with the same students for the duration of the trimester. In this model students are allocated randomly to as many as five different project teams, depending on the year level. In some instances, particularly in the early years there is scope to negotiate alternative membership. Such evolving team formation allows students to experience working with, and learning from at least fifteen other students over the unit’s duration. This emulates the reality of professional practice where project managers may find themselves working collaboratively on project ‘teams’ that are dynamic, diverse and where membership is not always of one’s choosing.

4. The provision of feedback channels that engages students in sustained self and communal reflection

Embedded within the model are a number of feedback channels that support the development of self and communal reflection. The channels include a two-way channel between teaching staff and students, and multi-level students-to-students channel. In the former the channel consists of the following interactions:

• Weekly feedback to the whole class at the project update sessions on the most and the least useful from the weekly submissions. Selected submissions are de-personalised and analysed with all students; students are required to reflect on their products in order to ‘assess’ themselves against the analysed ones. At the later stages of the unit students are invited to exercise their professional judgement and propose marks for their comparable submissions.

• On-the-spot feedback at the project meetings either when the progressive outcomes are worked out or when students present their final products to the wider audience.

• On-the-spot feedback at consultations where teaching staff and individual project teams discuss specific aspects of project or team performance. Each student team is responsible for preparing the agenda and are required to post/share their reflection (minutes of the consultations) with their relevant team members in their discussion forum.

• All students have access to a self-and-peer evaluation (SPE) tool, which allows students to reflect on their contribution to their team’s leadership, team performance and project outputs.

The multilevel students-to-students feedback channel consists of the following:

• Written feedback generated at project presentations, which is available on request if students would like to reflect on their abilities to convey concepts and ideas, and also on their presentation skills.

• Written feedback, as part of the self and peer evaluation, is available on request if students would like to reflect on their leadership and teamwork skills.

5. The provision of structured opportunities for students to develop and use ‘professional’ judgement that is explicitly induced and recognised

As previously mentioned, the whole learning environment, including the assessment model, is designed to support the intentional development of judgement. This is done through carefully structured opportunities where the development of judgement can be progressively practised and where awareness can be brought to bear on those judgements as they unfold. As Knight (2006, p.439) argues, judgements are more trustworthy when they are:

1. observations of several slices of practice, 2. in different settings and at different times, 3. by different observers… 4. …[made by those] who are trained in making judgements and 5. …with reference to known and agreed criteria.

In many respects the structured opportunities overlap with the preceding four characteristics. The development of students’ professional judgement starts when they are initially engaged in either individual or collaborative negotiations around the scope, objectives and deliverables for all projects. This is an iterative process as the progressive nature of negotiations provides an opportunity to adjust their judgement as they develop a deeper understanding of the projects and when more information is available to students.

As was described under point four, another major contributor to the development of students’ judgement are the weekly examples of professional judgement applied by teaching staff when they analyse individual or collaboratively produced project products. In the early stages of the unit, staff, with minimal student involvement, mainly perform the analysis. As the unit progresses the balance shifts and students become more actively involved in the analysis. In the final stages students are expected to analyse exhibited submissions, to reflect on their submission and exercise (and defend) their judgement by ultimately proposing a value, or mark, on their project. Staff then assesses their submissions, and the marks either confirmed or adjusted with explicit feedback given as to any difference between their judgement and that of the teaching staff.

In this model self-and-peer evaluation (SPE) is regarded as the major tool in the development of professional judgement. The SPE is an essential component and students must participate in self-and-peer evaluation in order for the marks to be finalised. The tool requires students to reflect upon a number of critical components related to the performance, management and leadership of each project and to actively apply judgement to their contribution and their peers. In addition to the comments on contribution, SPE requires students to evaluate their contribution in terms of a value or mark, and to assign a value to their team members. The comments and proposed marks are then analysed by teaching staff, compared with other evidence and, if they coincide, the proposed marks are awarded. In a case when the SPE does not coincide with staff’s observations and available evidence, students are required to explain the difference in judgement. All collaborative projects are self-and-peer evaluated and a total of 35% out of 100% can be awarded in this way.

Discussion

The shift to an integrated model of assessment, underpinned by practice, has not been without its challenges. When originally introduced in 2007, the shift proved confrontational to students and teaching staff. For students, the model as outlined here was markedly different to the assessment regimes of their other units, and of their secondary schooling. In particular students were accustomed to more traditional linear models, where one assessment finished before the next one commenced; the non-sequential and overlapping nature proved disorientating for some and pushed time and workload limits. However qualitative comments as part of an ongoing evaluation provide some positive insights. One student, for example, stated that ‘small integrated assignments and constant assessment is much better than a few large assignments’ (Student 2/SRM281 Project Management 2, 2008).

The ever-changing student teams within the confines of one unit created management and performance issues that are normally more contained in conventional team or group work practice. Positive reflections were a minority of the overall experience, but provide some intelligence, which can be built upon. For example, this student found this mode of team work beneficial, noting, ‘the practical side of approaching the assignments, and doing the assignments in groups, and the changing of the groups in each instance so we got to work with different people [was good]’ (Student 4/SRM281 Project Management 2, 2008). While another student reflected on the benefit of working with a broader range of students than they would have normally, ‘group work and getting to work with various members of the class [was beneficial]’ (Student 8/SRM261 Contract Administration 2, 2008). Several students alluded to the benefits of being assessed by their peers, and being able to exercise a judgement of their team members performance against criteria: ‘I also loved the fact that we would be assessed as to how we performed in the eyes of our team mates, and the people that didn’t perform properly could be assessed as such (quite liberating!)’ (Student 9/SRM281 Project Management 2, 2008). An ongoing challenge is to engender an openness and positive predisposition to such assessment activities and thinking, and to better manage students’ expectation with regard to the particular approach adopted in this stream.

Less contentious was the authenticity reflected in the range of assessment activities, suggesting that students appear more naturally drawn to assessment that attempts to mirror practice. In singling out aspects of the unit that students were favourable towards, this comment is reflective of students’ positive attribution, ‘creating our own contract, and administering it throughout various stages of development’ were felt to be of value (Student 6/SRM261Contract Administration 2, 2008). Another student noted in his professional portfolio (Student 21, 2009) ‘In reflection, I conclude that weekly PJ’s [Professional Journals] are a very useful management tool; allowing me to reflect, review and control my professional development on a weekly basis. I would strongly recommend this procedure to other management students and management professionals.’ This sense of control was echoed by yet another student in the context of negotiating aspects of the assessment criteria and in the opportunity to develop and use professional judgement that is explicitly recognised, ‘Having negotiated the assessment criteria for each project is much better than the assessment criteria imposed on us’ (Student 11/SRM281 Project Management 2, 2008).

There are some parallels with the informal feedback from teaching staff. The teaching staff who contributed to the delivery of the units were more familiar and comfortable with traditional assessment practice. In practice this meant that the Unit Chair drove the pedagogical agenda but acknowledges that in order for the model to be sustainable, all teaching staff need to have greater engagement and ownership than is currently the case. That said, early results are promising. We have seen an upward trend in our ‘Student Evaluation of Teaching of Units’ results. Of particular interest is that students, who are now in their second project management unit, and second experience of a ‘like’ model, rated their experiences more positively than their first experience. The stream continues to reflect upon students’ comments and refine and enhance along the way.

Conclusion

This paper described a model of integrated assessment, underpinned by key theoretical considerations from practice theory. Critical to the model was a view of assessment that was grounded in the professional world of project managers, that connected students to each other around meaningful activities, that provided structured opportunities for students to make judgements and receive judgements of others, and where that judgement had an impact. Future research directions will enable us to sample numerous rich data sources that the units are generating, including students’ feedback via the university student evaluation, self and peer reflections and evaluations, and access to students’ professional journals. It is hoped that by doing so within and across year levels we might be able to contribute positively to a range of pressing questions: the extent to which such a model actively contributes to the development of judgement for our graduates, and the extent to which attention to embodied learning challenges our assumptions about assessment (and learning) are two that are worthy of further empirical work.

Acknowledgement

The authors would like to acknowledge the useful comments received in an earlier version of the paper, presented at the ATN Assessment Conference, University of South Australia, Nov. 2008.