Computer-supported collaborative learning in the medical workplace: Students' experiences on formative peer feedback of a critical appraisal of a topic paper

Abstract

Background: Medical workplace learning consists largely of individual activities, since workplace settings do not lend themselves readily to group learning. An electronic Learning Management with System Computer-Supported Collaborative Learning (CSCL) could enable learners at different workplace locations to discuss personal clinical experiences at a distance to enhance learning.

Aim: To explore whether CSCL-enabled structured asynchronous discussions on an authentic task has additional value for learning in the medical workplace.

Methods: Between January 2008 and June 2010, we conducted an exploratory evaluation study among senior medical students that were engaged in clinical electives. Students wrote a Critical Appraisal of a Topic paper about a clinical problem they had encountered and discussed it in discipline homogeneous subgroups on an asynchronous forum in a CSCL environment. A mixed method design was used to explore students’ perceptions of the CSCL arrangement with respect to their preparation and participation, the design and knowledge gains. We analysed the messages recorded during the discussions to investigate which types of interactions occurred.

Results: Students perceived knowledge improvement of their papers. The discussions were mostly task-focused. The students considered an instruction session and a manual necessary to prepare for CSCL. A high amount of sent messages and a high activity in discussion seem to influence scores on perceptions: ‘participation’ and ‘knowledge gain’ positively.

Conclusion: CSCL appears to offer a suitable environment for peers to provide formative feedback on a Critical Appraisal of a Topic paper during workplace learning. The CSCL environment enabled students to collaborate in asynchronous discussions, which positively influenced their learning.

Introduction

For medical students, learning in the workplace is primarily an individual activity. For these students, work-based learning might nevertheless benefit from collaborative learning, which has been shown to stimulate critical thinking, deeper-level learning and shared understanding, thereby fostering active knowledge construction (Dewiyanti 2005; Kreijns et al. 2005; Van der Meijden 2005). The logistical problem of organising collaborative learning activities for students dispersed over different training locations might, at least partly, be resolved by the use of computers to virtually connect learners and facilitate collaborative learning at a distance. This can be achieved by an electronic Learning Management System (LMS) using Computer-Supported Collaborative Learning (CSCL) which supports different formats for distance learning. Studies of CSCL applications in medical classroom settings have demonstrated quite good instructional effectiveness (Levin & Ben-Jacob 1998; Panikkar et al. 1998; Walker et al. 1998; Devit & Palmer 1999), but comparisons with traditional classroom formats have thrown the added value of CSCL arrangements into doubt (Lieberman et al. 2002; Duque et al. 2006; Smith et al. 2007; Evans et al. 2008). CSCL arrangements in workplace learning might have additional value by enabling students to share clinical experiences, and use these experiences as a starting point for deep learning. There is, however, a paucity of research evidence about the use of CSCL in workplace learning in medical education. One study demonstrated knowledge improvement when CSCL was used to enable students to solve a theoretical clinical case in a controlled clinical workplace environment (De Wever et al. 2008), but essentially, we know very little about the effectiveness of CSCL in workplace learning. We, therefore, conducted an exploratory evaluation study using a mixed method design to explore whether or not CSCL adds value by facilitating collaborative learning from clinical experiences in the workplace. We sought students’ perceptions of the usefulness of a CSCL arrangement designed specifically for this study and measured students’ interactions and activities during CSCL-enabled discussions. We were interested in answering the following research questions:

1. How do students perceive the instructional quality of the CSCL arrangement, particularly with regard to preparation for, participation in, the design of and knowledge gains due to the CSCL arrangement?

2. How do students interact and what activities do they engage in, during a structured discussion of an authentic task in the CSCL arrangement?

Table

Practice points

CSCL is a promising format for formative peer feedback on CAT papers. Students actively engaged in discussion report high subjective scores on their participation and on knowledge gain. Students’ subjective perceptions on participation and knowledge gain seem to be associated with the number of sent messages and the level of discussion activity.

Methods

Participants

The study was conducted between January 2008 and June 2008, and between January 2010 and June 2010 in one teaching hospital (Maxima Medical Center, Veldhoven, The Netherlands) on medical students in the last year of the 6-year undergraduate medical curriculum of Maastricht University, which consists of two 18-week electives. For this study, 35 students were invited, whereas 27 students were agreed to participate. The participants received informed consent before the start of the study and were free to withdraw their cooperation at any time. Participants were allocated to homogeneous subgroups of three students, based upon the discipline of their elective.

As a general part of the elective, students are required to write a Critical Appraisal of a Topic (CAT) paper about a self-selected clinical problem they have encountered during the elective. The CAT paper should be written in a structured way, by presenting research evidence, aetiology, diagnosis, prognosis, management strategy and follow up in relation to the clinical problem (Sauvé et al. 1995; Parkes et al. 2009). In this study, it was the first time students discussed their CAT papers in an asynchronous forum of a CSCL environment. Discussion took place in homogeneous subgroups of three students on electives in the same discipline.

The CSCL arrangement

The CSCL environment used DOKEOS, an open-source LMS (www.dokeos.com) which can create online learning communities and enables delivery of instructional resources, communication and collaboration. The CSCL arrangement consisted of four consecutive phases. Phase 1: the participating students attended a face-to-face session in which they were instructed about the CAT assignment and the use of DOKEOS. To enable the subgroups to familiarise themselves with the CSCL environment, all students were provided with a manual, a time schedule for practice sessions for each subgroup and a pass-word and log-in code to access DOKEOS. The subgroup members were asked to schedule a practice session immediately after the instruction session. Phase 2: each student wrote a CAT paper and uploaded it to a ‘drop-box’ on DOKEOS. Phase 3: the subgroups discussed the CAT papers written by the three group members on the asynchronous discussion forum of DOKEOS. The discussion was structured in three prescribed steps:

1. discuss the strengths and weaknesses of the process of literature search;

2. discuss the strengths and weaknesses in the design and the execution of the research used to solve the clinical problem;

3. evaluate the quality and discuss the amount of evidence in the research used to solve the clinical problem.

The subgroups had to complete their discussions of the CAT papers within a 2-week period. It was up to the author of the paper to decide when the paper had been discussed properly. Phase 4: the students could revise their papers.

Study design

We used a mixed method study design to explore students’ perceptions of the educational value of the CSCL arrangement. The students were asked to rate questionnaire items on a five-point Likert scale (1 = absolutely disagree; 5 = absolutely agree) and take part in a semi-structured interview. Each interview lasted 30–45 min and was recorded by an electronic recording device. The questionnaire and the interview addressed the same four outcome measures in relation to the CSCL arrangement: ‘Preparation’, ‘Participation’, ‘Design’ and ‘Knowledge gain’. We used SPSS 16.0 to calculate descriptive statistics (mean, SD) for the questionnaire per subgroup, and the interviews were qualitatively analysed by the researcher (Baarda et al. 2001).

In order to address the second research question, ‘How do students interact and what activities do they engage in during discussions enabled by the CSCL arrangement?, we analysed the discussion messages that had been filed automatically by DOKEOS. We counted the overall number of messages, and the percentages of messages per subgroup. To find out how students interact, we assigned the messages to one of the following categories: ‘Message to the author of the CAT paper’, ‘Reply message from author to discussant’ and ‘Message between the two discussants’. To find out in what activities students engage, we assigned each message to one of the seven categories of the Rainbow system for collaborative problem solving activities in CSCL discussions (Baker et al. 2007): outside activity; social relation; interaction management; task management; giving opinion; giving argumentation and broadening/deepening. The Rainbow system is designed to measure the extent to which students are engaged in interaction. For each category and for each subgroup we calculated the percentage of activities.

Results

In this section, the results on the outcome measures ‘Preparation’, ‘Participation’, ‘Design’ and ‘Knowledge gain’ are described. In every outcome measure, the questionnaire score is described first and then all interview results are described. Every student participating in this study returned the questionnaire and was interviewed. Further on, students’ interactions and activities during the CSCL discussion are described and combined with students perceptions.

Students’ perceptions of the quality of preparation for participating in the CSCL arrangement

Table 1 shows that the quality of their preparation for the CSCL arrangement was rated positively by the students (mean 3.6, SD ± 0.48). The interviews showed that students considered the instruction session helpful to familiarise them with the CAT task. The students did not make use of the offered opportunity to practise working in the CSCL environment. The instruction manual was considered useful and used to look up the internet address, review the assignment, and navigate DOKEOS. It is interesting to notice that students said they appreciated the instruction session and used the manual, but did not use the opportunity to practice with the CSCL arrangement.

Table 1.  Mean scores (±SD) on the four outcome measures of students’ perceptions of instructional quality of the CSCL arrangement on a 5-point Likert scale (1 = absolutely disagree; 5 = absolutely agree)

	Preparation	Participation	Design	Knowledge gain
Subgroup 1	3.2 (±0.42)	4.2 (±0.08)	3.9 (±0.22)	3.9 (±0.23)
Subgroup 2	3.1 (±0.19)	3.9 (±0.14)	3.4 (±0.29)	3.7 (±0.29)
Subgroup 3	3.6 (±0.35)	3.9 (±0.46)	4.0 (±0.70)	3.8 (±0.39)
Subgroup 4	3.8 (±0.73)	3.5 (±0.30)	4.0 (±0.30)	3.6 (±0.48)
Subgroup 5	3.8 (±0.00)	4.0 (±0.30)	4.4 (±0.00)	4.0 (±0.29)
Subgroup 6	3.7 (±0.94)	3.8 (±0.71)	4.1 (±0.10)	3.7 (±0.47)
Subgroup 7	3.5 (±0.17)	3.6 (±0.30)	4.0 (±0.22)	3.8 (±0.23)
Subgroup 8	3.6 (±0.67)	3.8 (±0.46)	3.4 (±0.25)	3.3 (±0.22)
Subgroup 9	3.3 (±0.17)	3.9 (±0.25)	3.8 (±0.08)	3.5 (±0.55)
Mean	3.6 (±0.48)	3.8 (±0.42)	3.7 (±0.47)	3.6 (±0.49)

Students’ perceptions of their participation in the CSCL arrangement

Students’ participation in the CSCL arrangement consisted of engaging in forum discussions about their CAT papers by posting opinions, comments and feedback. The questionnaire score in relation to participation was 3.8 (±0.42; Table 1). The students indicated that the CAT papers were discussed adequately during the forum discussions. Students also indicated that participation was time consuming and added to their work load during the elective. Although it was not obligatory for students to write and discuss the CAT paper, the students felt that participation in the CSCL arrangements had a positive effect on their learning. The students considered 2 weeks an acceptable period for the forum discussion.

It seems quite surprising that students evaluated their participation positively despite the time pressure and additional work load.

Students’ perceptions of the CSCL design

Students rated the quality of the CSCL environment 3.7 (±0.47) on a five-point scale (Table 1). They thought that the layout of DOKEOS was clear and the tools were user friendly. They appreciated having time to think about their contribution to the discussion before posting a message. They also thought that the asynchronous discussion forum was arranged and structured conveniently, and was appropriate for the CSCL arrangement and the specific CAT task. The students did not think that the discussion, as an extra element of the CAT task, interfered with other tasks during their elective.

Students’ perceived knowledge gains after participating in the CSCL arrangement

The students rated perceived knowledge gain as 3.6 (±0.49; Table 1). They thought participating in the asynchronous CSCL discussions had improved their knowledge. Students reported gains in knowledge with regard to scientific reasoning as a result of analysing a clinical problem, generating search criteria, executing an appropriate search strategy and using databases for literature search. Students said they had learned to evaluate and critically appraise a research article with regard to: journal impact factor, study population and sample, selection criteria, statistics, epidemiology and conclusions.

The students also remarked that the asynchronous CSCL discussions led to further knowledge gains in medical content knowledge which they needed to solve the clinical problem. After the discussion, students revised their CAT paper with respect to both content and structure. In summary, students reported knowledge gains and improvement of their CAT paper as a result of CSCL with asynchronous discussions.

Students’ interactions and activities during CSCL-enabled discussions

Students’ interactions as recorded in DOKEOS consisted of 316 messages (Table 2), with an average of 35 messages per subgroup. The average number of messages posted during the discussion of one CAT paper was 11 to 12. Individual students posted three to four messages per discussion. Classification of the messages showed, on the average, that 60% of all messages were sent to the author of the paper under discussion, 37.1% were replies by the author, and 2.9% concerned messages between the two discussants. Remarkable is that the more messages were sent, the more discussion took place between the three students of the subgroup.

Table 2.  The number of messages sent by participants, and the percentages of sent messages categorised in: messages sent to author; reply messages from author; messages between discussants, for each subgroup and in total

	Number of messages	Messages sent to author (%)	Reply messages from author (%)	Messages between discussants (%)
Subgroup 1	41	53.6	41.5	4.9
Subgroup 2	68	67.6	30.9	1.5
Subgroup 3	53	64.1	32.1	3.8
Subgroup 4	20	90.0	10.0	0.0
Subgroup 5	52	57.7	36.5	5.8
Subgroup 6	14	35.7	64.3	0.0
Subgroup 7	24	54.1	41.7	4.2
Subgroup 8	10	40.0	60.0	0.0
Subgroup 9	34	50.0	47.1	2.9
Total	316	60.0	37.1	2.9

The categories of activities during the discussions are shown in Table 3. Task-focused activities (categories 4, 5, 6 and 7) accounted for 81.9% of all subgroup activities, with 80.3% of the activities being discussion activities (categories 5, 6 and 7). Non-task-focused activities (categories 2 and 3) accounted for 13.5% and outside activities (category 1) for 4.4% of the activities. Subgroup 2, in which the most messages were sent, showed discussion activity only. Subgroups 4, 6, 7, 8 and 9, in which a low number of messages were sent, showed no broaden and deepen activities.

Table 3.  Percentages of students’ discussion activity as recorded in the LMS and classified in one of seven categories of collaborative problem solving activities in CSCL discussions (Rainbow system, see text)

	Collaborative problem solving activities
		Non-task-focused activity		Task-focused activity
	Category 1. Outside activity (%)	Category 2. Social relation (%)	Category 3. Interaction management (%)	Category 4. Task management (%)	Category 5. Giving opinion (%)	Category 6. Giving argumentation (%)	Category 7. Broaden and deepen (%)
Subgroup 1	2.4	24.4	7.3	4.9	31.7	24.4	4.9
Subgroup 2	0.0	0.0	0.0	0.0	61.7	26.5	11.8
Subgroup 3	2.0	9.4	0.0	0.0	56.6	26.4	5.6
Subgroup 4	0.0	15.0	0.0	0.0	80.0	5.0	0.0
Subgroup 5	11.5	11.5	1.9	1.9	32.7	30.9	9.6
Subgroup 6	14.3	14.3	7.1	0.0	42.9	21.4	0.0
Subgroup 7	0.0	12.5	12.5	4.2	58.3	12.5	0.0
Subgroup 8	0.0	10.0	10.0	10.0	40.0	30.0	0.0
Subgroup 9	11.7	8.8	2.9	0.0	52.9	23.5	0.0
Mean	4.4	10.4	3.1	1.6	50.6	24.0	5.7

Students’ perception combined with interaction and activities during CSCL-enabled discussions

For every subgroup, students’ perceptions were combined with the number of messages and with students’ activity on the forum. On students’ perceptions: ‘participation’ and ‘knowledge gain’ subgroups 1 and 5 showed high scores, with a high number of messages sent and replied by all students. Furthermore, these subgroups showed discussion activity in every category. Although, discussion activity was merely task-focussed, where ‘giving opinion’ occurred most frequently, with lower frequencies for ‘giving arguments’ and ‘broaden and deepen’. Subgroups with a low score on perceptions: ‘participation’ and ‘knowledge gain’ showed no broaden and deepen activities. Other students’ perceptions, ‘design’ and ‘preparation’ showed no clear influence on sent messages and discussion activities.

Discussion

We explored whether or not a CSCL arrangement offered added value to the learning of students from their clinical experiences in the medical workplace by conducting a mixed-method design.

The students gave a positive judgement of their preparation for participation in the CSCL arrangement by an instruction session and a manual. They said it helped them get used to the CSCL arrangement and tasks, and to execute the tasks properly. This can be confirmed by other studies (Komoroski 1998; Schellens & Valcke 2005; De Wever et al. 2009) which have measured whether or not preparation for CSCL influenced task execution. These studies showed that it saves time to prepare participants in CSCL by informing them of the design and technical aspects of the CSCL environment, such as access, use of tools and tips and tricks on how the forum works, because preparation can solve many questions and problems before students actually engage in the CSCL tasks.

Students gave high ratings on their participation in the forum discussion. Considering students thought participation required considerable effort in terms of time and work, and even though the task was not obligatory it is remarkable that students showed quite high scores on participation in the discussion. This may be associated with the fact that the subgroups were homogeneous as to speciality as well as to the tight time schedule.

Students’ perceptions of the design of the CSCL arrangement were positive. This is encouraging in light of findings that students’ positive perceptions can mediate the meaningfulness and the effectiveness of classroom learning arrangements both traditional and CSCL ones (Laurillard 2002; Morrison 2004; Dewiyanti 2005; Driscoll 2005). This study showed that structuring students’ tasks and the forum discussion may be important to achieve meaningful and effective learning.

Students reported knowledge gains after interacting in the CSCL arrangement, with improvements in medical content knowledge as well as in scientific reasoning. According to students, this resulted in improvements in both content and structure of their CAT papers. Knowledge gains were also found in studies of the instructional effectiveness of CSCL in classroom settings (Levin & Ben-Jacob 1998; Panikkar et al. 1998; Walker et al. 1998; Devit & Palmer 1999). Like our study, these studies only reported students’ self-perceived knowledge gains.

The average number of messages on the discussion forum was considered to be sustainable by the researchers. However, there is no research-based standard available for a sustainable number of messages when students interact. According to the messages sent, discussion was merely focussed on the author and activity was highly task-focussed. However, on subgroup level, scores were scattered. When students’ perceptions were combined with the sent and replied messages during discussion, and with students’ activity in discussion, students scores on ‘participation’ and on ‘knowledge gain’ showed influence on the discussion. Subgroups with high ratings on participation showed a discussion between all subgroup members and all subgroup members were highly active in discussion. Since the main purpose of the CSCL arrangement was to enable students to share, discuss and elaborate on their personal experiences as a way to promote effective learning (Laurillard 2002), these positive ratings are an important indication of the effectiveness of the CSCL arrangement.

One of the limitations of this study is that the outcomes, apart from the analysis of the discussions, were solely based on students’ perceptions. Further studies will be needed to obtain evidence from additional sources to confirm the findings, such as evidence from knowledge tests that students’ knowledge actually improved after their participation in CSCL. Furthermore, this study was conducted on one specific task, a structured discussion of a CAT paper. CAT is an important task in workplace learning, for students learn to analyse a clinical problem, which arise from their own experience, in an evidence-based manner (Sauvé et al. 1995; Parkes et al. 2009). Another limitation is that this study was conducted in one teaching hospital. This facilitated the researcher to provide students with a proper instruction about the CSCL arrangement and allowed direct control of the LMS if students experienced problems or asked questions about the system. Due to these limitations, the results of this study may not be generalisable, to other tasks and/or other settings. Further on, it could be argued that an open forum discussion would stimulate the unrestricted debate. However, since former studies have demonstrated that a structured discussion in CSCL is indispensable, we finally decided to ask the students to concentrate upon three key-items of a CAT (Baker & Lund 1997; Hron et al. 2000; Fischer et al. 2002; Hirsch et al. 2004). A final limitation is that no control group was included in the study.

We recommend further studies of the transfer of knowledge to be applied in different settings of the medical workplace. Research should identify knowledge transfer resulting in behavioural change of medical students to benefit patient care. Research should also examine which educational format is most suitable for which CSCL task.

Furthermore, in the field of content-analysis, further research should determine the number of messages that is sustainable for a reliable outcome and examine which type of content analysis system is suitable for which educational method.

Conclusion

Based on the results of this study, we conclude that CSCL appears to be a promising format to enable students to provide formative feedback on each others CAT paper. Subjectively, students expressed positive opinions about their experiences with a CSCL environment in terms of participation and knowledge gain. Analysis of the CSCL discussions showed that students who are engaged actively in discussions, show high scores on their participation and on knowledge gain. Students’ subjective perception on participation and knowledge gain seems to be connected with sent messages and discussion activity. Further studies are recommended to investigate whether students’ subjective positive perceptions can be confirmed by objective evidence from additional sources.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the paper.