Technology supported learning (TSL) for formative assessment

Abstract

Online environments offer educators a new medium to deliver teaching and learning material — one which can bring new and exciting ways of learning and an alternative to traditional teaching techniques. These new techniques can provide solutions to the demands of a changing environment, allowing flexibility for learning from home or work and the ability to cope with a widening variety of backgrounds and qualifications. The difficulty comes in finding ways to apply the new technologies to a learning process with proven educational benefit. This paper presents a case study involving an undergraduate module on distributed simulations in design within the Architecture and Product Design Department at the University of Wolverhampton. It discusses the planning, implementation, delivery and evaluation of online tasks for teaching Virtual Reality Modelling Language (VRML). This is supported by background research on the impact of using e-learning strategies for formative assessment. The evaluation of this experience is based on the analysis of students’ reflections and feedback, in particular how beneficial that experience was towards improving their understanding of the subject and achievement in their work. It concludes with future actions and ideas for improvement based on the analysis and synthesis of the implemented online tasks.

Background

Online environments can potentially offer students many possibilities for enriching the learning process, compared to traditional learning strategies. According to French et al. (1999), students can use online environments for more communication possibilities, collaboration, opportunities, interaction with their lecturers/classmates and access to a wider range of learning resources. Graff (2004) outlines that there are numerous clear theoretical advantages of online instructional methods. Firstly, they provide flexible learning, meaning that students can progress at their own pace, and secondly they provide the facility for student-centred learning, making students responsible for their own learning. Finally, material can be made available on demand from anywhere at any time, provided the learner has the facility for taking advantage of such system. These new techniques can provide solutions to the demands of a changing environment, allowing flexibility for learning from home or work and the ability to cope with a widening variety of backgrounds and qualifications. The difficulty comes in finding ways to apply the new technologies to a learning process with proven educational benefits (Allen, 1998).

Assignments provide learners with opportunities to discover whether or not they understand, if they are able to perform competently and to demonstrate what they have learnt in their studies. Also the feedback and grades that assessors communicate to students serve to teach and motivate (Thorpe, 1998). Furthermore, opportunities to explore new assessment methods and styles are now afforded by, for example, timed online exams, group projects and peer/self assessment (O’Reilly and Newton, 2001).

Online formative assessment has many advantages over traditional classroom assessment. Zakrzewski and Bull (1999) stated that students can take the assessment at any time, they can take it repeatedly and it can provide instant feedback that helps remedy weaknesses in their learning abilities. They also indicated that student anxiety could be reduced if they take the formative assessment before summative tests. According to Buchanan (2000), a web-based formative assessment strategy can improve student learning interest and student scores. Formative assessment allows students to assess their own progress and understanding. Clariana (1997) noted that formative assessment designs should be able to engage student attention, engender student commitment to self-evaluation and enhance learning effectiveness.

Research studies have discussed evaluations of what types of formative assessment strategies would work best. Buchanan (2000) argued that ‘repeat the test’ is an important design of web-based formative assessment strategy. However, he noted that this design strategy must be implemented in conjunction with the functions of ‘provide with no answer’ and ‘instant feedback’ so that the web-based formative assessment will be more effective.

Wang et al. (2006) refer to studies made by Federico (2000), Kraus et al. (2001), Buchanan (1998) and Terrell (2002) when looking at improvement of the effectiveness of e-learning, and developed an e-learning environment that integrated formative assessment strategy. The purpose was to discover whether the benefit of e-learning environments varies among learners with different learning styles. Three questions were explored in their research. First, do learning styles and formative assessment strategies affect student learning achievement? Second, what kind of formative assessment strategy can be built into the e-learning environment to facilitate student learning? Finally, what kind of learning style best suits the e-learning environment? The results of this study indicated that both learning style and formative assessment strategy significantly affect student achievement in e-learning. This suggested that both formative assessment strategy and learning styles should be taken into account in the design of e-learning environments. They also suggested that successful teachers tend to be those who are able to use a range of teaching strategies and a range of interaction styles, rather than a single rigid approach to teaching and learning.

Su et al. (2005) focused on both quantitative and qualitative data to investigate how instructors and students perceive the importance of online interactions. They discussed various forms of interaction, including learner-instructor, learner-learner, learner-content, learner-interface, learner-self interaction, and vicarious interaction. This study concluded that students generally perceive interaction as an effective means of learning which varies according to differences in personality or learning style as well as the level of online interactions. Based on these findings, the implementation of e-learning strategies for formative assessment would be enhanced by providing online access to information, and training through tutorials and instructions based on the widest possible range of learning interactions.

TSL for formative assessment

Interactive distributed simulations is a 30-credit level two module which is taught to a class of 20–30 students. In the year of this study there were 22 students with various backgrounds enrolled on this module. The principal aim of the module is to introduce students to the area of interactive distributed simulations and their implications for the design process. The module investigates the use of 2D and 3D virtual worlds that can be delivered over networked environments. Students learn VRML to assist them in creating 3D models that can be delivered across the web. By successful completion of the module students will be able to show:

• knowledge and understanding of the principles of technologies in relation to human needs and an awareness of potentials and limitations in their future developments;

• ability to apply knowledge and understanding to the solution of qualitative and quantitative problems of a familiar and unfamiliar nature;

• produce schematic layouts and select appropriate methods and materials for a given situation with due regard to regulations and economic factors.

The understanding and practice of this subject is crucial to the students as they have to complete the required assessment to pass this module. This assessment is made up of two coursework components worth 40% and 60% respectively: component one consists of an in-class timed assessment (Phase Test) and component two consists of the development of an interactive 3D web based simulation. The delivery of this module normally consists of a three-hour weekly lecture which includes PowerPoint presentations, class discussions and tutorial exercises.

Planning and implementation

The planning process is one of the most important stages in implementing any e-learning strategy as it saves time by identifying problems before they arise. Graham et al. (2000) emphasised the factors that are to be considered during the planning stage. These can be best approached as a series of considerations that will help in developing a clearer image of the parameters. Table 1 was constructed using Graham’s method to include all the necessary planning and delivery data.

Using Table 1 online tasks were developed and implemented using the online learning facilities (Wolf) within the University of Wolverhampton. They were a formative assessment with no contribution to the marking criteria. They were developed to enhance and consolidate students’ learning and understanding of the subject to achieve the best results in their phase test. They were uploaded onto Wolf on a weekly basis for students to download and complete in their own time. Students were encouraged to reflect on their experience and each other’s using the posted message facility within Wolf and also to provide feedback at the next face-to-face lecture. Any problems that were encountered were then discussed and resolved prior to the students’ engagement with the next task. In order to support this need a number of online tutorials/tasks were devised and uploaded for students to carry out. By completing these activities students were able to:

• define the concepts of different nodes within the VRML language, such as text node, DEF node, inline node, anchor node, lights, fog, sounds and LOD node;

• define irregular geometry node to create complex objects in VRML;

• define irregular extrusion node to create complex paths for different faces;

• define terrain modelling node to create different terrains;

• explore how to control navigation of the VRML world by the user;

• evaluate model constraints and position using appropriate calculation methods;

• analyse the use of different nodes together in one file;

• analyse how to change the visualisation of a VRML world by exploring the background node;

• analyse the texture coordinate node to map materials on different surfaces;

• apply knowledge and understanding of the use of VRML by generating 3D objects and copying them in different positions.

Table 1 Planning and delivery of TSL

	Considerations	Implications
Preliminary
Subject	Develop a 3D virtual world using VRML.	Understanding and applying the syntax of VRML.
Medium	Use of Notepad software. Use of VRML browser. Links to VRML specifications website. Online self-assessment.	Access to VRML browser. Large specification website, taking longer time to navigate.
Students
Stage	22 level 2 students from various undergraduate courses within the Architecture and Product Design Department.	All taken 3D modelling modules before, however little on VRML programming. Assume most have some experience of directed learning.
Access	Some will access the material from home, some from the library, but most from the university computing rooms. Notepad is a standard package. VRML browser can be easily downloaded and installed free.	Some home computers are below required specifications, hence students may not be able to complete all the tasks at home.
IT skills	Mixed abilities of students.	Extra tutorial sessions may be required.
Learning styles	Mixed learning styles.	A range of material and activities to suit different approaches.
Learning
Relation to module	Online tasks will run for the first five weeks of the module. Individual tasks will reflect the theme of the subject discussed in lecture. Students are required to complete one task per week and provide feedback and reflection.	Task material needs to be developed and uploaded on to Wolf at the appropriate time ready for students to download and complete. Include instructions together with images of task outcomes.
Learning outcomes	Students to investigate the use of 2D and 3D virtual worlds that can be delivered over networked environments.	Outline learning outcomes section within the online tasks. Include methods for students to measure their progress. Provide means for students to submit feedback and reflection.
Assessment	Made up of two coursework components worth 40% and 60% respectively.	Successful completion underpins the learning outcomes and supports students in achieving the best possible results in the required assessment.
Development	Depending on student feedback, future tasks may vary.
Resources	The use of Wolf system to support the delivery of this TSL.

To evaluate the effectiveness of online tasks on students’ learning, Su et al. (2005) pointed out that there is a need to understand which instructional techniques and activities can promote interaction in online education. Summarising key points, giving feedback and instructor participation in class discussion were widely used within the teaching of this module. Such activities indicated that instructor-learner interactions were in play. These activities aided students’ reflective thinking. Students were working together to solve problems and were also involved in class discussions to critique each other, helping to establish a rich interaction between them that indicated learner-learner interactions. Students were asked to explore and then analyse the content of the online tasks in order to achieve the required results. It was also necessary to interlink contents between the online tasks as they were dependent on each other. This helped students to interact with the content of the taught subject, indicating learner-content interactions.

Reflection and evaluation

There is a need to understand the way students used the online tasks and the factors that enhanced their learning. The purpose of obtaining student feedback is to gain information that would help to initiate and continue improvements in interactive online teaching and learning (Thorpe, 1998).

A survey of the students was conducted following their completion of the phase test. The aim was to gather feedback from them on their perception of features that were aiding or distracting from their learning in the online environment, more specifically the relevance of online tasks in preparation for summative assessment. 14 out of 22 attempted the online tasks, 13 (92%) completed the feedback questionnaire. Based on the overall results the majority of students (92%) indicated that the online tasks were very beneficial and helped them in preparing for the phase test. As one student said, ‘online tasks made us think about the use of VRML code, rather than just copying from tutorials’. Students found it necessary to relate what was taught within the lectures to what was asked from them via the online task, making them rely on their analysis knowledge and skills rather than just copying notes. This was facilitated by the way the tasks were designed and by the fact that they were able to assess their performance immediately, reflect on what they had done wrong, and rectify it with the tutor’s and other colleagues’ help and advice. The questionnaire also covered other aspects of development, delivery and support of the online tasks. The overall feedback was very positive, as students gave strong comments about the benefits they experienced. The students’ response is summarised in Table 2. Students were asked to fill in their recommendations of what can be done to improve such a facility. They expressed that the instructions for the online tasks could be less restrictive and also suggested adding more examples and images. The results of this questionnaire clearly support the authors’ conclusions that such online activities/tutorials help students in their learning and self-dependency. The final results of the students, when compared to the previous year’s results, also confirm these claims.

Analysis and synthesis of results

The development and delivery of the online tasks as a formative assessment to enhance and prepare students’ learning for the summative assessment was successful and has shown positive results. However, evaluating these online activities poses a series of issues at various levels. Three aspects are of particular significance: the first is evaluation of learning, the second is evaluation of the students’ performance, both in terms of participation and activities effectively carried out, and the third is interaction and collaboration between students to reflect on each other’s experiences.

’For any evaluation to be successful it is important that you see its context as one of both improvement and effectiveness, depending upon the level being carried out. The evaluation process should consider all features of the learning events that may promote or hinder progress.’

(Jolliffe et al., 2001.)

Throughout this online experience the students have shown significant improvement in their learning of the taught subject. The combination of feedback from students and comparison of results with the previous year leaves little doubt that online teaching has improved students’ learning. However, the challenge was to convince students to participate fully with such activities. The problem was that students felt there was no incentive to engage, and that created additional pressure on the tutor to continuously encourage and motivate students to take part. Therefore, future online tasks ought to include elements of summative assessments which would then ensure student participation. ‘Students must not feel that e-learning is just afterthought tacked on and therefore strong links with the subject aims must be evident. In the same contrast, by building a formal assessment element into the e-learning task, all students will seriously engage with these tasks and not only the interested ones’ (Graham et al., 2000).

Although most students carried out the online tasks and discussed and reflected on their experience within the face-to-face sessions, they did not fully utilise the forum posting facility to reflect on their ongoing experience. Only a limited number reflected on each other’s work, which thus limited the benefits that they could have gained. In order to ensure students’ interactions and online reflections, future online tasks should incorporate a requirement for students to provide a reflection on individual tasks and each other’s. ‘Sharing understanding and supporting each other in an open unstructured forum suggests that deeper social and learning communication networks are developing among learners’ (O’Reilly & Newton, 2001). ‘In order to stimulate motivation, e-learning should be accessible and involve high levels of interaction between learners and resources, learners and teachers as well as learners and learners’ (Graham et al., 2000).

Conclusion

This paper has investigated the planning, implementation and evaluation of using TSL for formative assessment. Students’ reflections and feedback were compiled during their learning experience and then analysed to gain an understanding of the impact of such methods on their learning. The overall results of using e-learning for formative assessment have provided some support for the argument that online systems for students’ teaching, learning and evaluation can be reliably implemented. However, a number of questions, problems and anomalies have been uncovered which need to be resolved before the potential benefits that were originally envisaged can be fully realised.

Table 2 Summary of the questionnaire results

Questions	Strongly agree 1	Agree 2	Strongly disagree 3
Purpose and objectives of the tasks were made clear	39%	61%	0
Content and presentation of the tasks was appropriate	54%	39%	7%
Tasks were well structured and easy to follow	46%	54%	0
Tasks method used was appropriate and effective	54%	46%	0
Tasks satisfied my personal objectives	46%	54%	0
Tasks were useful/worthwhile	69%	31%	0
Module materials were helpful	69%	31%	0
Sufficient support and feedback from the tutor	62%	38%	0
Tasks were helpful for assessment preparation	85%	15%	0