How to configure blended problem based learning–Results of a randomized trial

Abstract

Background: Problem based learning (PBL) may be successfully complemented by Blended Learning approaches. However, the precise effect of combining different eLearning components in PBL-courses is yet unclear.

Aim: This study aims at comparing the effects of the different combinations of three interactive components (Wiki, Chat, and, Interactive Diagnostic Context – a diagnostic related link collection to expert sources) on learning, aspects of group process, and individual learner satisfaction.

Methods: A mixed-method study addressed all possible combinations of the three components. Tests and controls were assigned by group randomization. Hypotheses were tested by the Friedman- and the Mann–Whitney U test.

Results: The results prove Wiki to positively influence the ratings given by students to the course, whereas no evidence supports the expected advantages of Chat. The Interactive Diagnostic Context led to only few significant effects concerning the diagnostic approach and general score.

Conclusion: Blended PBL (bPBL) profits best from supporting asynchronous communication (i.e. Wiki). Synchronous communication components and hypothesis-driven information retrieval do not yield further improvements. Thus, bPBL should concentrate on technically fostering the group process and avoid overloading the course configuration by other eLearning components.

Introduction

Problem based learning (PBL) and blending learning (BL) have successively enriched teaching in medical schools over the last decade. Previous study has shown that both approaches can be successfully combined (Taradi et al. 2004; Derntl et al. 2006; Kim et al. 2006; Donnelly 2006; De Leng et al. 2006; Williams 2006). Still, it is unclear as to which combination of eLearning components enhances the effectiveness of PBL the best. This question has not been addressed previously by comparative studies.

This study analyzes three components: Wiki, Chat, and, interactive diagnostic context (IDC). We assume that:

• The eLearning components of Wiki and Chat lead to a more efficient level of communication, foster the collaborative effect of PBL, enhance the student's satisfaction, and improve the test results in comparison to the absence of this variable (Table 1).

• IDC leads to a more specific approach during the diagnostic process, improves the student's satisfaction, makes Blended PBL (bPBL) more realistic, and improves test results in comparison to the absence of the variable.

Table

Practice points

bPBL is an adequate alternative to cPBL bPBL should concentrate on technically fostering the group process bPBL best profits from supporting asynchronous communication (here: Wiki) Chat and IDC do not yield further improvements of bPBL if a Wiki is available and cannot compensate for its absence

Background

Problem based learning

PBL is a method of teaching influenced strongly by constructivism (Savery & Duffy 2001). Students form small groups and solve a presented problem without the normative guidance of a teacher providing all the necessary information. Since classical (i.e. non-blended) problem based learning (cPBL) was first used in the context of medical education at McMasters University in the late 1960s, the approach has gained widespread use. Likewise, the PBL methodology has evolved (McPhee 2002). A widely applied cPBL format, referred to as “Seven-Step process,” was developed at Maastricht University. It defines the following seven steps (Schmidt 1983):

1. Case presentation: The PBL case is presented and unknown terms are explained.

2. Problem definition: The group defines the fundamental issue.

3. Brainstorming: The students collect ideas.

4. Formation of hypothesis: The results of the brainstorming are formed into a working hypothesis.

5. Defining educational objectives: The students define a detailed agenda to “gain more profound knowledge of the processes forming the crux of the problem” (Schmidt 1983).

6. Self-study: The students acquire the necessary knowledge by themselves.

7. Synthesis: The results are presented within the group and the case is revisited based on the insights achieved.

Table 1.  Overview of the results

Hypothesis	Wiki	Chat	IDC
Better learning effect	✓	−	−
Better communication	✓	−	Ø
Better collaborative effect	✓	−	Ø
Higher amount of satisfaction	✓	−	−
More specific diagnostic approach	Ø	Ø	✓/−
More realistic bPBL	Ø	Ø	−

Note: Notice that the table refers to each eLearning component in every combination in comparison to non-component groups.

✓, hypothesis proven; −, hypothesis not proven; ø, does not apply.

PBL has been proved equal to lecture-based education in a university setting concerning general success (Albanese & Mitchell 1993; Vernon & Blake 1993; Enarson & Cariaga-Lo 2001; Michel et al. 2002; Dettmer et al. 2002; Newman 2003; Kawai et al. 2007). Advantages of this learning approach can be found in the fostering of communication skills, teamwork, and learning strategies (Mennin et al. 2003; Hofgaard Lycke et al. 2006). On the other hand, more resources such as staff and infrastructure are required by cPBL in comparison to ordinary teaching (Wood 2003). Furthermore, not all students are able to cope well with cPBL (Wieking 2005).

Blended learning

Blended learning (BL) combines (“blends”) different educational resources in order to optimize the educational experience for any learner. A main issue of BL is the combination of face-to-face and online learning. The usage of online-learning tools in higher education is resource efficient (Sancho et al. 2006), leads to similar test results (Chen & Zimitat 2004; Sancho et al. 2006), enables a more individualized learning (Pahinis et al. 2007), and is enjoyed by the students (Higgins & O’Keeffe 2004; Pahinis et al. 2007).

Blended PBL

cPBL and BL approaches have been successfully combined (Taradi et al. 2004; De Leng et al. 2006; Derntl et al. 2006; Donnelly 2006; Kim et al. 2006; Williams 2006; Woltering et al. 2009). In the following pages, we refer to the combination of cPBL and BL as blended problem based learning (bPBL).

Existing approaches of bPBL enhanced the classical face-to-face PBL with separate computer-based or web-based applications serving as supporting tools, referred to as eLearning components in the following. For example, the PBL case can be presented by multimedia (De Leng et al. 2006). Collaborative work may be supported by an online discussion board (Oliver & Omari 1999; De Leng et al. 2006). It is possible, to integrate research recourses of the world-wide-web into cPBL scenarios (Oliver & Omari 1999). The respective combination of eLearning components of a bPBL course will be referred to as the bPBL course configuration.

Previous studies have focused on the practicability of cPBL, BL, and bPBL in general (Albanese & Mitchell 1993; Sancho et al. 2006; Woltering et al. 2009). The question how the course configuration should be set in order to adequately configure an online-learning environment for bPBL has not been sufficiently answered. Similarly, the combination effect of eLearning components used is still unclear. To the best of our knowledge, there is no study addressing which special eLearning components should be included in a bPBL scenario and which bPBL course configuration serves the best.

Aim of the study

Different course configurations may be chosen in order to form a bPBL course. No evidence is given in which way a combination of single eLearning components influences the impact of the bPBL course.

This study aims at comparing and analyzing the effects of different course configurations on the learning effect (self-assessment score), on features of the group process, and on the individual learner satisfaction. Our study should yield implications for advising the choice of eLearning components for bPBL scenarios in medical education.

Curricular context

The medical program at RWTH Aachen University follows a 6-year curriculum, which has been redesigned as a “Modellstudiengang” (“model-curriculum”) with the beginning of the winter term 2003/2004. Instead of following the traditional layout of strict separation of pre-clinical and clinical as well as strictly subject-bound teaching, a sequence of multidisciplinary modules (e.g. “heart and circulation” or “growing and pediatrics”) supports a spiral-shaped learning process focusing on systems and organs. Nearly all modules contain cPBL tutorials. The tutorials follow the Maastricht “Seven-Step” approach (Schmidt 1983; Wood 2003). Two of these cPBL tutorials have been transformed into BL (bPBL) scenarios by supporting the cPBL by selected eLearning components. The project developed through a cooperative effort of the clinicians involved, the local Institute of Medical Informatics, and medical students. A detailed description of the bPBL project has been published previously (Woltering et al. 2009).

Material and methods

ELearning components

This study addressed the possible effects of the combinations of eLearning components on the bPBL module. In order to draw clear and statistically sound conclusions, this study was limited to three components in every possible combination. The central features of interest of PBL under investigation were the components specifically related to PBL didactics in terms of collaborative learning and constructivism. The selected components should:

1. cover a wide range of component types and

2. support complementary aspects of interaction.

Collaboration and communication can be synchronous or asynchronous, allowing immediate reaction or time-shifted continuation of work, respectively. Wiki was chosen as an exemplary asynchronous collaboration tool. Synchronous collaboration was best supported by a Chat component. The third component chosen for this study is referred to as interactive diagnostic context (IDC): the component was designed to provide additional diagnostic information upon demand.

Wiki

A Wiki is a hypertext system that can be read and easily edited by its users within the web browser. It enables asynchronous communication through written texts. Using a simple mark-up, any word/phrase of a text can be declared a keyword, and can be automatically linked to another Wiki page, which contains additional information on the topic. Different users can edit the same page. The editing of any text may take place at any time and as often as wished. The Wiki system provides a detailed history of changes. A Wiki enhances the sharing of and collaborative working on the concepts (Cubric 2007). Cubric points out the importance of tutor feedback upon the successful use of a Wiki in BL. In PBL, tutors are essential elements of the course concept and therefore present. Given the collaborative merits (Himpsl 2006, p. 75) of a Wiki and the demand for collaboration inherent in the PBL approach, it appeared important to further investigate the effect of this component on bPBL.

Chat

A Chat is an online tool that enables written real-time communication. Whereas the information cannot be presented as densely or as well organized as in the Wiki, the Chat provides its users with immediate reactions and a rapid response to partners. This oral-like concept of communication (Storrer 2001) can be mediated in writing and may add to the collaborative process required by PBL. Apart from the collaborative effects, a Chat may also function as an information source for the individual learner (Singh 2003). For these reasons, we chose a Chat as one of the variables.

IDC

The IDC provides additional information relevant for the diagnostic process. The information is only available for specific requests (e.g. MRI information are only presented on the learner's demand). Thus, the IDC fosters a hypothesis-driven information search. The IDC contains a collection of links to web sites containing case-related references. This process simulates some aspects of a clinical order entry system, where special examinations can be requested in order to get the corresponding laboratory or physical findings. The (hypothesis-driven) problem-solving behavior supported by the IDC corresponds to the constructivist's approach of PBL, where “the processes forming the crux of the problem” (Schmidt 1983) are more important than the solution itself.

Layout of the course

The PBL course consists of two tutorials separated by a few days. During the first tutorial, the students execute the first five steps of the seven-step approach (i.e.: case presentation, problem definition, brainstorming, formation of hypothesis, and defining educational objectives). During the interval between the tutorials, students are free to work on the 6th step, whenever they like. Finally, the second tutorial gives the final synthesis (step 7).

The bPBL scenario uses the same approach. All the different course configurations follow the Maastricht seven-step PBL (Schmidt 1983). There are two essential differences:

1. the bPBL tutors and all members of each bPBL group are granted access to a virtual, group specific workspace, which is set up for them by the local learning management system,

2. the first tutored group-meeting of the bPBL is substituted by a group cooperation based on the virtual workspace, which can be freely accessed by the group members.

Thus, each group member executed the first five steps of the Maastricht-seven-step approach individually using the online workspace. The necessary information and explanations were emailed previously to each student. Whether the student preferred to work alone or wished to collaborate with group members could be individually chosen. Collaboration was limited to the virtual workspace. Each subgroup (see below) of roughly 8 to 9 students had their personal virtual workspace.

Both, the 6th and 7th steps were unchanged to the cPBL routine. Although tutors are needed, they can work from anywhere. Furthermore, 50% less room time is required by bPBL in comparison to cPBL.

The virtual workspace contained the case vignette, an interactive history component, an interactive medical testing component with a limited budget and – depending on the group – the three components Wiki, Chat, and IDC as pointed out below.

The history component enabled the students to assess a large number of medical issues referring to the case in question. If not applied selectively, the assessment would take a long period of time.

The medical testing component enabled the student to perform a large number of tests (labwork, imaging, genetics, etc.). The number of tests performed was limited by a financial budget.

Participants

The participants of the study were drafted from the Aachener Modellstudiengang Medizin (Medical School of the RWTH Aachen University). All medical students within the academic year (n = 237) were included, as well as all PBL tutors on duty (except of the author of the study). The evaluation was set up in the context of the multidisciplinary module “growing and pediatrics” during the summer term 2008.

Study design

In order to find out about the effect of the eLearning components, eight different bPBL course configurations were compared with each other and a control group.

A total of 237 students participated in the study. The cPBL control group consisted of 99 students divided into six groups, and 138 students participating in bPBL (Figure 1). Since all the medical students of a year were included, this was the maximum number of participants which could be included without mixing different years with different exposures to PBL during the course of the curriculum.

Figure 1. Distribution of participants: the participants (n = 237) are divided into cPBL (n = 99) and bPBL (n = 138).

The 138 students in the bPBL branch formed eight groups with eight different course configurations. Each group had the same basic online environment (see below). The three eLearning components (Wiki, Chat, and IDC) were distributed with the eight groups in a way that formed every possible combination including a negative control that only consisted of the basic online environment (Figure 2). The basic online environment is the virtual online workspace equipped with the case vignette, a virtual interactive medical history, and a tool for querying diagnostic tests (for example different CT scans). All bPBL groups had the exact same learning environment except for the three randomly distributed components Wiki, Chat, and IDC.

Figure 2. Distribution of course configurations: the figure shows the distribution of the eLearning components Wiki, Chat, and IDC within the bPBL groups.

Figure 3. Evaluation flow chart: the chronological order of the evaluation-interventions is shown.

It is important to notice that each of the 14 groups split up into two equally sized subgroups for the PBL tutorial. This guarantees a PBL-adequate group size. The data of both subgroups using the same bPBL configuration was pooled.

This layout was chosen in order to analyze influential factors, and to enable assessment of various combinations of factors. Sometimes, the combination of two beneficial factors can cause a negative impact due to interference (Schorer & Raab 1999).

Instruments

The evaluation of the different course configurations adopted:

1. a questionnaire (for cPBL and bPBL students, and for tutors),

2. a self-test

3. an instrument for self-assessment, and

4. structured interviews.

These instruments have been specifically developed for this evaluation in order to tailor the evaluation instrument to the needs of adequately addressing the hypotheses.

Questionnaire

The cPBL and the bPBL participants answered the same questionnaire. The only difference was that strictly bPBL-specific items have been excluded from the cPBL questionnaire. These items would have caused confusion within the non-bPBL group because of reference to eLearning components.

The questionnaire consisted of 35 questions (the self-test questions are included within the following numbers) for cPBL and 55 questions for bPBL (Appendix) to be answered freely (eight questions cPBL, nine question bPBL), by checking preformed categories (eight for classical, 11 for bPBL, and rating on a five-point scale (28 items classical, 45 items bPBL, and sometimes one question includes several sub-questions with one item each). Further, there was room for write-in comments. The items covered communication within the participants (ability to communicate, density, lastingness, organization potential, time needed), the collaborative effect (cooperation), the participants’ satisfaction (grade), test and learning results (difficulty, preparedness, and self-test), the role of the tutor, the quality of the user manual for the online-environment, the diagnostic process (selectivity, costs, and adequacy), and realism of the course (brainstorming, clarification of terms, hypothesis formation, documentation of results, and working on tasks). Appendix shows a translated bPBL questionnaire as an example. It has been custom made for this evaluation.

Self-test

The questionnaire both for cPBL and bPBL students contained the same five self-test questions (questions 51–55 in the Appendix). These questions referred to objectives that should be learned by the students during the PBL course. By monitoring these objectives, it was possible to determine if students had gained the necessary basics during the course.

Self-assessment

The learners’ self-assessment concerning abilities and knowledge linked to the PBL case (Movement physiology, genetics in pediatrics, diagnostic of ataxias, differential diagnosis, and diagnostic approach) took place three times: (1) before the course, (2) after the course reflecting upon the self-assessment before the course (question 49 in the Appendix), and (3) after the course (question 50 in the Appendix) referring to the current state assessing the same items each time. It consisted of five items linked to a five-point scale.

The three instruments (1) questionnaire, (2) self-test, and (3) self-assessment have been pre-tested by a group of six medical students not involved in the study. All independent variables have been primarily assessed with the questionnaires for students and the self-assessment.

Interview

The purpose of the semi-structured interview was to gain further detailed information and to verify the results of the questionnaire. Therefore each interview result was compared to the questionnaire filled out previously by the same person. A single interview was performed with each of the 17 volunteers from cPBL and bPBL groups. The same standard questions were used in every interview. These questions referred to the same item complexes as did the questionnaires for students without asking literally the same question. Furthermore, the evaluation process itself was also addressed such as the length of the process, time granted, and understandability.

Tutors

The evaluation of the tutors was performed via specific questionnaires. It consisted of 11 questions to be answered: one by open comments, one by checking preformed categories, and seven by rating on a five-point scale. In addition, the participants could enter open comments. This questionnaire targeted the workload for the tutors, their preference of being physically present during the first tutorial, and their perception of the student's group dynamics.

Randomization

The students were assigned to learning groups, which (apart from singular exchanges) remained stable during the course of the medical curriculum and were the basis of the timetable scheduling. As a consequence, the group assignment could not be changed for this study. Therefore, block randomization was implemented: The groups were arranged into a random sequence by a random sequence generator (online available at: www.random.org/sequences) and then linked to a prefixed test plan in order to guarantee a blinded distribution. A person not involved in the study (Alexandra Kühl) monitored this procedure.

Statistical methods

Data analysis was based on a Friedman test and a nonparametric Mann–Whitney U test due to the data being not normally distributed. The level of significance was established at an error probability of α = 5%.

Results

Learning effect

The self-tests addressing learning objectives show no differences between groups with a Wiki to other bPBL groups, and compared to cPBL groups as well (Table 2). Lower scores are achieved by groups utilizing Chat compared to bPBL groups without Chat. The same holds for the group equipped with only IDC. The presence of Chat in the online environment goes along with a decrease in the ability to solve the case correctly (Table 2).

Table 2.  Comparison of learning effects: p-values

Mann–Whitney U test: two-sided significance	Difficulty of the case	Acceptance of the case's difficulty	Feeling to have covered everything	Preparedness for the exam	Right diagnosis (%)	Number of right answers (maximum five) in the self-test
Wiki
Group only with Wiki vs. bPBL negative control	0.044	>0.1	>0.1	>0.1	>0.1	>0.1
Group only with Wiki vs. all other bPBL groups	0.001	>0.1	>0.1	0.093	>0.1	>0.1
All groups with Wiki vs. all groups without Wiki	0.002	>0.1	>0.1	>0.1	>0.1	>0.1
Chat
Group only with Chat vs. bPBL negative control	>0.1	>0.1	>0.1	>0.1	>0.1	>0.1
Group only with Chat vs. all other bPBL groups	>0.1	>0.1	>0.1	>0.1	>0.1	>0.1
All groups with Chat vs. All groups without Chat	>0.1	>0.1	>0.1	0.006	0.070	0.082
IDC
Group only with IDC vs. bPBL negative control	0.012	0.002	>0.1	0.064	>0.1	0.049
Group only with IDC vs. all other eGroups	0.001	0.007	>0.1	>0.1	>0.1	>0.1
All groups with IDC vs. All groups without IDC	0.004	0.020	0.098	>0.1	>0.1	>0.1

Note: Underlined values signify α < 5% and italic values signify α < 10.

Items addressing the students’ reflection of the learning effect yielded the following results (Table 2): Groups utilizing Wiki show lower marks on the case difficulty and higher marks concerning the students’ feeling of being prepared for the exam than those without access to Wiki. The presence of Chat in the online environment corresponded with lower marks concerning the preparedness for the exam (Table 2). The IDC groups show lower marks on the acceptance of the cases’ difficulty, the preparedness for the exam and the feeling to have covered all relevant aspects.

Self-assessment

The Friedman test yields the following results: For both cPBL and bPBL, all five self-assessment items change significantly from the measurement before the course (pre start), after the course reflecting upon the self-assessment before the course (retrospective), and after the course (post course) (Table 3). The participants using Wiki rate all items significantly different between all three measurements. For all non-Wiki groups, the item concerning movement physiology does not change significantly.

The same result applies to Chat. While all items for the groups with Chat change significantly, the movement physiology does not change for all non-Chat groups.

Concerning the IDC the opposite observations are demonstrated. All items change significantly for the non-IDC groups, whereas the movement physiology does not change for the IDC groups.

The combination of a Wiki and a Chat results in a significant change of all five self-assessment items. This also applies for the combination of a Wiki, a Chat, and IDC. All other combinations show significant changes in fewer cases (Table 4).

Table 3.  Self-assessment of all bPBL groups: Mean(SD) on a five-point scale from 1to 5

Item	Pre start	Retrospective	Post course
Movement physiology	2.17 (0.895)	2.22 (0.801)	2.57 (0.877)
Genetics in pediatrics	2.19 (0.943)	2.23 (0.974)	2.72 (0.942)
Diagnostic of ataxias	1.42 (0.544)	1.80 (0.822)	2.91 (0.935)
Differential diagnosis of movement disorders	1.73 (0.766)	2.15 (0.848)	2.88 (0.931)
Structured diagnostic approach	2.36 (1.019)	2.95 (1.035)	3.34 (0.935)

Note: All numbers were rounded to the hundredth.

Table 4.  Friedman test for the H0-hypothesis: The self-assessments (I) pre start, (II), retrospective, and (III) post course are equal

	Movement physiology	Genetics in pediatrics	Diagnostic of ataxias	Differential diagnosis of movement disorders	Structured diagnostic approach
bPBL groups	0.001	0.000	0.000	0.000	0.000
cPBL groups	0.000	0.000	0.000	0.000	0.000
Wiki groups	0.007	0.000	0.000	0.000	0.000
Non-Wiki groups	0.125	0.000	0.000	0.000	0.000
Chat groups	0.003	0.000	0.000	0.000	0.000
Non-chat groups	0.162	0.000	0.000	0.000	0.000
IDC groups	0.099	0.001	0.000	0.000	0.000
Non-IDC groups	0.013	0.000	0.000	0.000	0.000
Wiki and Chat group	0.021	0.044	0.001	0.003	0.006
Wiki and IDC group	0.264	0.368	0.047	0.039	0.116
Chat and IDC group	0.174	0.368	0.002	0.001	0.001
Only Wiki group	0.943	0.085	0.004	0.097	0.050
Only Chat group	0.472	0.066	0.048	0.048	0.050
Only IDC group	0.097	0.012	0.002	0.005	0.012

Note: Underlined values signify α ≤ 5%.

Communication

In general, items addressing aspects of communications are rated higher by bPBL groups featuring a Wiki than by groups without a Wiki: This applies to the items referring to density and lastingness of the communicated information, and to an item referring to the organization potential delivered through the communication as well (Table 5).

Table 5.  Comparison of communication: p-values

Mann–Whitney U test: two-sided significance	Online communication	Adequacy of the time needed to operate communication	Organization of work flow via communication	Density of the communication within the program	Longevity of information communicated within the program	Comparison of the organization achieved in bPBL to cPBL
Wiki
Group only with Wiki vs. bPBL negative control	>0.1	>0.1	0.066	>0.1	>0.1	0.001
Group only with Wiki vs. all other bPBL groups	>0.1	0.005	0.005	0.032	0.040	0.000
All groups with Wiki vs. all groups without Wiki	>0.1	0.063	>0.1	>0.1	>0.1	0.007
Chat
Group only with Chat vs. bPBL negative control	0.005	>0.1	>0.1	>0.1	>0.1	>0.1
Group only with Chat vs. all other bPBL groups	0.035	>0.1	>0.1	>0.1	>0.1	0.042
All groups with Chat vs. All groups without Chat	>0.1	0.014	0.007	>0.1	0.062	0.019
IDC
Group only with IDC vs. bPBL negative control	>0.1	>0.1	>0.1	>0.1	>0.1	0.047
Group only with IDC vs. all other eGroups	>0.1	>0.1	>0.1	0.058	>0.1	0.078
All groups with IDC vs. all groups without IDC	>0.1	>0.1	>0.1	>0.1	>0.1	>0.1

Note: Underlined values signify α < 5% and italic values signify α < 10.

Communication density addresses the compactness of the communicated information (How effectively is an information chunk delivered by the communicator?). The lastingness of communication measures the subjective time a communicated item is available to the addressed communication partner.

Collaboration

The expected improvement of collaborative effects due to a Wiki or a Chat could not be proven (Table 6).

Table 6.  Comparison of collaboration and satisfaction: p-values

Mann–Whitney U test: two-sided significance	Cooperation within the group	Overall grade in German school grades (1 –6 )
cPBL vs. bPBL	>0.1	0.000
Wiki
Group only with Wiki vs. bPBL negative control	>0.1	>0.1
Group only with Wiki vs. all other bPBL groups	>0.1	0.001
All groups with Wiki vs. all groups without Wiki	>0.1	>0.1
Chat
Group only with Chat vs. bPBL negative control	>0.1	0.026
Group only with Chat vs. all other bPBL groups	0.000	0.063
All groups with Chat vs. all groups without Chat	0.001	0.000
IDC
Group only with IDC vs. bPBL negative control	0.068	0.002
Group only with IDC vs. all other eGroups	>0.1	0.070
All groups with IDC vs. all groups without IDC	>0.1	>0.1

Note: Underlined values signify α < 5% and italic values signify α < 10.

Student satisfaction

The overall bPBL satisfaction is inferior compared to cPBL. Comparing the bPBL group, which used a course configuration exclusively equipped with a Wiki, to all other bPBL groups shows an increase of the satisfaction with the course in the presence of a Wiki. Students, whose online environment contained a Chat, are less satisfied with the bPBL course.

The bPBL group equipped only with IDC is less satisfied with the course than all other bPBL groups. No difference can be found by comparing all IDC-groups with the IDC-negative control (Table 6).

Diagnostic approach

Wiki-groups score lower concerning the accuracy of the diagnostic process than those without a Wiki. The ability to gain diagnostic information in a structured manner is rated lower by all groups equipped with Chat compared to those without Chat. Evidence is presented by the comparison of the bPBL group only with Chat and the bPBL negative-control, as well as with all other bPBL groups, and by the comparison of all bPBL groups with Chat and those without.

The IDC groups are more aware of the costs of their diagnostic steps than the bPBL groups without (Table 7).

Table 7.  Comparison of diagnostic approach: p-values

Mann–Whitney U test: two-sided significance	Amount of selectivity during the diagnostic process	Influence of the costs of a diagnostic step	Importance of the diagnostic steps being adequate
Wiki
Group only with Wiki vs. bPBL negative control	>0.1	0.056	>0.1
Group only with Wiki vs. all other bPBL groups	0.009	0.034	0.041
All groups with Wiki vs. All groups without Wiki	0.002	>0.1	0.077
Chat
Group only with Chat vs. bPBL negative control	>0.1	0.065	>0.1
Group only with Chat vs. all other bPBL groups	>0.1	0.099	>0.1
All groups with Chat vs. All groups without Chat	>0.1	>0.1	>0.1
IDC
Group only with IDC vs. bPBL negative control	>0.1	>0.1	>0.1
Group only with IDC vs. all other eGroups	>0.1	>0.1	>0.1
All groups with IDC vs. all groups without IDC	>0.1	0.013	>0.1

Note: Underlined values signify α < 5% and italic values signify α < 10.

Realism

bPBL equipped with a Wiki achieves higher scores on the items regarding the realization of the steps of the seven-step concept. This addresses the participant's ability to follow the seven-step approach with the online environment. No evidence supports the expected positive effects of the IDC (Table 8).

Table 8.  Comparison of realism: p-values

Mann–Whitney U test: two-sided significance	Online brainstorming	Online clarification of terms	Online hypothesis formation	Online documentation of results	Online working on tasks
Wiki
Group only with Wiki vs. bPBL negative control	>0.1	>0.1	>0.1	>0.1	>0.1
Group only with Wiki vs. all other bPBL groups	>0.1	>0.1	0.058	0.021	0.008
All groups with Wiki vs. all groups without Wiki	0.071	>0.1	0.017	0.014	0.018
Chat
Group only with Chat vs. bPBL negative control	>0.1	>0.1	>0.1	>0.1	>0.1
Group only with Chat vs. all other bPBL groups	>0.1	>0.1	0.054	>0.1	>0.1
All groups with Chat vs. all groups without Chat	>0.1	0.028	>0.1	>0.1	>0.1
IDC
Group only with IDC vs. bPBL negative control	>0.1	>0.1	>0.1	>0.1	>0.1
Group only with IDC vs. all other eGroups	>0.1	>0.1	>0.1	>0.1	0.052
All groups with IDC vs. all groups without IDC	>0.1	>0.1	>0.1	>0.1	>0.1

Note: Underlined values signify α < 5% and italic values signify α < 10.

The findings above are supported by the free comments within the questionnaire and the interview results. It should be noticed that the original statements were made in German and have been translated by the author.

The participants stated that the communication worked well with the eLearning component Wiki whereas limitations applied to the Chat. Such comments included: The Wiki worked well regarding the exchange of findings and defining learning goals. and The Chat did not work because not every member of the group was able to join the session and it took for ever to operate. The students preferred asynchronous communication tools. Communication was pointed out as essential for the success of any PBL course.

The students were able to associate the eLearning components at hand within the online-learning environment with other resources not provided by the course configuration such as drug-information-sheets for instance. A typical comment was After being confronted with the eLearning environment I looked for different forms of Ataxia online and tried to differentiate between them.

Regarding the setting of bPBL some participants appreciated the replacement of the first course tutorial by the online-learning environment because of gain of freedom and the lack of disturbance while others missed the personal interaction. Typical responses included that bPBL is, as far as I am concerned, better because it is less time consuming and more effective and Personally, I prefer working in a small group for the sake of better communication. Mimic and gestures are important for communication and are missed out by the online environment.

The interview shows that the questionnaires were easy to understand and the time granted to fill them out was adequate (Table 9).

Table 9.  Item values on a five-point scale (1–5) as measured through the questionnaires: mean (standard deviation)

Item	All cPBL	All bPBL	Wiki	Chat	IDC
Cooperation	3.88 (0.916)	3.67 (1.139)	3.63 (1.202)	3.33 (1.116)	3.71 (1.107)
Overall grade (in German school grades: 1 –6 )	2.79 (0.979)	3.07 (0.971)	2.93 (0.897)	3.42 (0.970)	3.21 (0.950)
Online communication	ø	1.61 (1.153)	1.74 (1.195)	1.60 (1.196)	1.55 (1.060)
Online brainstorming	ø	2.49 (1.468)	2.75 (1.519)	2.36 (1.456)	2.58 (1.423)
Online-clarification of terms	ø	2.89 (1.364)	2.90 (1.376)	2.60 (1.335)	2.84 (1.288)
Online-hypothesis formation	ø	2.67 (1.316)	2.96 (1.283)	2.47 (1.295)	2.67 (1.233)
Online-documentation of results	ø	2.95 (1.484)	3.31 (1.380)	2.89 (1.456)	2.918 (1.418)
Online working on tasks	ø	2.71 (1.333)	3.02 (1.350)	2.62 (1.304)	2.65 (1.220)
Adequacy of the time needed to operate communication	ø	3.00 (1.319)	3.25 (1.266)	2.72 (1.199)	2.96 (1.228)
Organization of work flow via communication	ø	2.35 (1.153)	2.49 (1.120)	2.09 (1.165)	2.22 (0.994)
Density of the communication within the program	ø	2.64 (1.302)	2.80 (1.212)	2.45 (1.153)	2.46 (1.284)
Longevity of information communicated within the program	ø	2.26 (1.126)	2.31 (1.140)	2.06 (1.066)	2.13 (1.085)
Comparison of the organization achieved in bPBL to cPBL	ø	2.25 (1.289)	2.65 (1.426)	2.00 (1.286)	2.16 (1.172)
Amount of selectivity during the diagnostic process	3.59 (0.960)	3.47 (0.973)	3.16 (1.032)	3.45 (1.152)	3.55 (0.862)
Influence of the costs of a diagnostic step	2.48 (1.279)	3.64 (1.161)	3.47 (1.345)	3.62 (1.114)	3.93 (0.953)
Importance of the diagnostic steps being adequate	3.64 (1.029)	3.80 (1.034)	3.57 (1.207)	3.84 (1.118)	3.91 (1.031)
Difficulty of the case	3.39 (1.050)	3.37 (1.226)	2.98 (1.326)	3.29 (1.165)	3.71 (1.124)
Acceptance of the case's difficulty	3.44 (0.968)	3.47 (1.061)	3.45 (0.997)	3.53 (0.858)	3.22 (1.109)
Feeling of having covered everything	2.35 (1.048)	2.76 (1.133)	2.69 (1.079)	2.57 (1.191)	2.58 (1.068)
Preparedness for the exam	2.22 (1. 049)	2.44 (1.219)	2.44 (1.208)	2.11 (1.155)	2.30 (1.190)
Right diagnosis (in %)	75	92	91	87	91
Number of right answers (maximum five) in the self-test	3.81 (1.075)	3.96 (1.064)	4.02 (0.913)	3.80 (1.061)	3.91 (0.996)

Note: ø = does not apply. Underlined values signify scores > 3.8 of 5.

Discussion

Based upon the findings above, the hypothesis may be reevaluated as following:

Hypothesis-review

• The eLearning components of Wiki and Chat lead to a more efficient communication, foster the collaborative effect of PBL, enhance the student's satisfaction, and improve the test results in comparison to the absence of the variable.

  • Accepted for Wiki.

  • Rejected for Chat.

• IDC leads to a more specific approach during the diagnostic process, improves the student's satisfaction, makes bPBL more realistic, and improves test results in comparison to the absence of the variable.

  • Accepted concerning the more specific diagnostic process.

  • Rejected for satisfaction, realism and test results.

Study design

This study delivers a systematic randomized analysis of the effect of the eLearning components (Wiki, Chat, and IDC) on the learning effect (self-test, self-assessment, and preparedness for exams), on features of the group process, and on the individual learner satisfaction in detail.

Selection bias is minimized through the cluster randomization using a random sequence generator under surveillance of an uninvolved and thus neutral person.

The advantages of the chosen approach are the comparisons not only of bPBL groups with each other, but also the comparisons of the bPBL with the current standard of cPBL.

The return rate of 77.2% is satisfactory.

The findings are validated by triangulation by comparing the questionnaire results with the interviews and free comments.

Concerning the method of measuring the self-assessment, the advantage of the triple assessment (pre, retrospective, post) is that not only the subjective gain of knowledge and abilities is being shown, but also the change in the student's self confidence as well as the revaluation of their abilities once confronted with the problem. The advantage of the group layout and paired comparison lies in the ability to detect interferences or synergies between factors that may or may not be beneficial by themselves but cause a different effect if combined (Schorer & Raab 1999). These effects must be taken into consideration, in order to draw valid conclusions concerning the three tested eLearning components.

Effects

Wiki

The Wiki is able to enhance the subjective learning effect and seems to promote the student's self-confidence. The conclusions concerning the self-confidence were drawn, based upon the self-assessment and the preparedness for the exam.

The communication turns out to be denser and more lasting if a Wiki is included into the course configuration as pointed out by the questionnaire scores. This leads to a better realization of the seven-step concept and a higher organization potential of the online-environment. One could argue that this is also an important reason for the higher amount of satisfaction and motivation with the course delivered through the Wiki. Thus, motivation is confirmed by this study as an important factor (Dantas & Kemm 2008) for educational process.

Probably for the same reasons, students working with a Wiki considered the course easier and felt better prepared for the exam than those without Wiki.

Collaborative learning and constructivism are fundamentals of PBL didactics. Wiki helps demonstrate these concepts through its qualities as a dense and well-organized communication tool. Our study confirms the expectations regarding the merits of a Wiki (Zorko 2007) within BL.

Chat

Unfortunately the Chat server broke down several times during the evaluation period in spite of all prior testing. This led to a slow and sometimes malfunctioning Chat. In consequence, all Chat relevant results suffer from a negative bias of unknown strength.

The ability of the Chat to foster the subjective learning effect and promote the student's self-confidence is measurable, but smaller than the Wiki's.

The results of this study show no evidence of a Chat improving the communication within the online environment.

A stronger collaborative effect mediated by the Chat was not measurable.

IDC

IDC stimulate the subjective learning effect and promote the student's self-confidence. The effect is not as strong as by the Wiki.

Only the cost awareness hints to a more specific approach during the diagnostic process.

No evidence supports the thesis that IDC lead to a more realistic learning environment.

As expected, the IDC is able to foster the PBL participant's learning process and therefore adds to a more constructivist learning process.

Related work

To the best of our knowledge, no previous study has systematically addressed the effects of different configurations on bPBL scenarios. Related studies focused upon the general possibility of using one or the other configuration, respectively, upon the comparison of bPBL and other methods of teaching (Taradi et al. 2004; De Leng et al. 2006; Derntl et al. 2006; Donnelly 2006; Kim et al. 2006; Williams 2006; Woltering et al. 2009).

Limitations of the study

Limitations apply because of the technical problems that caused a restricted Chat function. Furthermore, and in spite of the satisfying return rate of the questionnaires, it is likely that those not showing up to the second tutorial which included the evaluation may be those most disappointed by the course. This being the case, negative votes would have been systematically left out.

Due to the practical restrictions of group size, it cannot be ruled out that significant findings may have been masked (Tables 2–8). Concerning the analysis of the self-assessment, the last seven comparisons in Table 4 (from Wiki, Chat, and IDC group to only IDC group) have a low power due to a restricted group size, whereas the other comparisons in Table 4 have a lower risk of accidently accepting the H0-hypotheses since a larger set of data could be used.

Furthermore, self-report measures may not be totally accurate. By using a wide spectrum of items as well as triangulating our measurements, we have tried to minimize any bias related to self-report.

The length of the questionnaires for students bears the risk of question fatigue. The interviews addressed the length of the questionnaires and found that participants approved the number of questions it contained. Group dynamics may have influenced the individual perception and evaluation behavior. Even by pooling the two subgroups into one set of data these effects cannot be excluded from the statistical analysis.

The large amount of statistical tests performed goes along with the possibility of 5% of the tests being wrong in average. However, this large number was necessary due to the detailed approach. No conclusion is based on one test result alone. Instead, trends of significant differences were confirmed.

Conclusion

Generally speaking, this study confirms the previous results (Taradi et al. 2004; Kim et al. 2006; Woltering et al. 2009) that bPBL proves itself to be an adequate alternative to cPBL within the seven-step concept. The slightly inferior outcome can be explained by the fact that all course configurations together form the basis of the bPBL data including the poorer equipped ones. Furthermore, the participants were not yet acquainted to the new format.

Woltering et al. (2009) confirm the findings in hand that bPBL can successfully substitute cPBL by documenting superior subjective learning gains, motivation, and satisfaction within a bPBL compared to cPBL course.

In order to follow the didactic concept of PBL effectively, communication within the groups is essential. This is shown by the reduction of the student's satisfaction in the absence of communication eLearning components as well as in the case of the Chat server's malfunction and by the interview. Apparently, the malfunction of an eLearning component leads to a higher amount of discontent than the general lack of the same instrument.

Therefore, this study contradicts the general view that the more learning components present are always more useful.

Motivation is another important factor (Dantas & Kemm 2008) for the success of any educational process. It is difficult to objectively measure motivation. Analyzing the tutors’ evaluation as well as those of the student's appears to demonstrate no general motivational difference between bPBL and cPBL exists. Within the bPBL groups, however, the broken down Chat has led to a certain level of frustration.

Concerning a bPBL course configuration it is generally advisable to enable focused and dense communication. Wiki by itself may already be sufficient. The complexity of the configuration should be limited primarily in order to help focus upon relevant tools and, secondarily, in order to minimize the risk of technical problems which form a relevant risk for the course. Following the seven-step approach works very well for bPBL. Since it is well established in cPBL it is recommendable to use this approach. Using tools similar to IDC enhances constructivist learning approaches and thus seems to be of advantage.

Summary

Communication is identified as one of the most important success factors for bPBL. Wiki could be proven to be superior to Chat and IDC regarding the influence upon the success of a bPBL course. Yet, Chat and IDC do not fulfill the expectations, in spite of not seeming to bring harm to the concept, and thus may be used in addition. They may enhance the success for some, if not for the average since every participant is free to use the course configuration in any way one likes. Generally speaking, a simpler course configuration is preferable over an overloaded configuration.

The comparison of Wiki and Chat concerning their role within the bPBL concept should be repeated in another study, in the absence of technical malfunction. Another question of research is how to further improve the communication tools themselves with respect to user friendliness, administrative, and technical recourses, as well as concerning the quality of communication.

Acknowledgments

We thank the team of the Institute for Medical Informatics of the RWTH, all cPBL/bPBL tutors, and all students participating in the evaluation but especially Ulrike Moeller and Gary Uremovich for reviewing this article.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of this article.

Appendix

Notice that this questionnaire was translated for the readers of this article. The original one being used during the evaluation is in German.

Questionnaire for students: bPBL

Dear students,

We kindly ask you to spare a few minutes of your time in order to evaluate the teaching concept you just tested. Please fill out the questionnaire in hand. Your data will be handled strictly anonymously.

It is essential for a valid measurement that you fill out the questionnaire by yourself.

Please return the questionnaire to your tutor.

Thank you in advance!