Enhancing postgraduate students’ technical skills: perceptions of modified team-based learning in a six-week multi-subject Bootcamp-style CS course

ABSTRACT

Background and Context

Objective

This study explores postgraduate students’ perceptions of the modified team-based learning instructional approach used to teach it and the extent to which the Bootcamp course improves their practical skills.

Method

In the beginning, participants (n = 185) were asked to rate their practical experience on the taught topics. At the end of ProgSD in S1 and Semester 2 (S2) during the Covid19 lockdown, respondents (n = 150 and n = 43, respectively) were surveyed on their perceptions of ProgSD. Wilcoxon test, Mann-Whitney U Test and Kruskal-Wallis test were used to analyse data. Thematic Analysis was used to analyse students’ comments.

Findings

Results showed that students’ practical experience significantly increased at the end of ProgSD with a p-value < 0.05. Median ratings of the modified TBL activities and perception of teamwork were positive overall, and many (66.5%) felt more confident about taking the courses in S2. Qualitative results supported these findings.

Implications

Findings have implications for practitioners facing the challenges identified here. Given the current pandemic, the authors suggest approaches to redesign face-to-face activities (tAPP) for online teaching and learning.

KEYWORDS:

• Team-based learning
• active learning
• computing science
• programming
• higher education
• bootcamp
• postgraduate education

Introduction

Team-based learning

Research has highlighted that active learning methods foster deeper conceptual learning of traditional teacher-centred methodology (Biggs & Tang, 2011). Team-based learning (TBL) is an active learning method devised by Larry Michaelsen, which is extensively used in medicine and mainly in small group teaching (Michaelsen et al., 2004). It is a highly structured learning design that incorporates individual formative feedback, peer interaction, and active peer management of the team interactions. It is a student-centred approach led by teacher instructions to promote active learning. Many empirically supported best practices of student-centred active learning are brought into a single strategy (Leupen, 2020). Four principles govern TBL (Michaelsen et al., 2004, 2008):

a) Team formation. Permanent and strategically formed and managed small teams from week one;

b) accountability is essential as each student is responsible for their own learning and contributes to the team in order to achieve a higher level of cognitive learning (Vasan et al., 2009)

c) in-class teams must foster interaction, promote team learning and development;

d) immediate feedback is particularly critical to reinforce student learning and enables team development. With TBL, the exams are no longer “end-of-unit exercises”. Instead, they take a preparatory role.

The TBL sequence starts with student preparation which requires individual pre-reading and studying out of the class. Stage 2 (Readiness Assurance) tests students’ knowledge of the pre-studied subject, first individually through individual readiness assurance tests (iRATs), then through team readiness assurance tests (tRAT) in the class. During this stage, teams can appeal if they believe the answer to a question is not correct. Furthermore, the instructor provides feedback tailored to the evidence of iRAT and tRAT results and clarifies any further misunderstandings. At stage 3, students start applying what they have learned through discussion of high-level concepts and problem-solving activities.

TBL has been used in other disciplines, including engineering, business, and computing science (CS) (Christensen et al., 2019; Diniz et al., 2019; Elnagar & Ali, 2013; Ghadiri et al., 2013; Lasserre & Szostak, 2011; Matalonga et al., 2017). There is a growing body of evidence suggesting that students achieve more academically with TBL than the traditional approaches to instruction (Reimschisel et al., 2017). A TBL review by Swanson et al. (2019) on the effect of team-based learning on content knowledge and comprehension found that it improves students’ end of course grades, test performance, and classroom engagement. It can improve the attendance rate (Inuwa et al., 2012) and improve the final exams scores more than the traditional lecture environment (El-Banna et al., 2020). TBL stimulates deep learning, fosters collaboration, teamwork, enhances student engagement and communication skills (Vlachopoulos et al., 2020). This instructional approach is about interactive small-group learning that fosters critical thinking. Furthermore, even when compared to a moderated structured active learning, TBL still provides greater learning (Ng & Newpher, 2020).

In CS, in particular, it has been found that this student-centred instructional approach has many advantages, including a high level of engagement, better academic performance for weaker students, enhanced programming skills and confidence in the ability to program (Elnagar & Ali, 2013; Kirkpatrick & Prins, 2015; Lasserre & Szostak, 2011; Rankin et al., 2007). This study will examine whether some of these advantages are observed in a new postgraduate course we created: Programming and Systems Development (ProgSD).

Our context in creating ProgSD

Increasing enrolments in Computing Science have created significant demands and challenges (Camp et al., 2017). In our school, postgraduate students coming from a variety of backgrounds come with different skills sets, which our lecturers are having trouble addressing. So in order to better serve our students given their diverse skills sets, we created a new program, Programming and Systems Development (ProgSD), that they will all go through together so we can standardise their knowledge. This will make it easier for our lecturers in future classes to know which skills they can rely on students having, which was the lecturers’ complaint. We hope that with lecturers’ better able to rely on specific skills, their classes will work better for students, and student performance will increase.

The course is offered to all our master’s (CS+) programmes: MSc Data Science (DS), MSc Computing Science (CS), and MSc Information Security (IS). All students on these programmes are assumed to have a CS or software development undergraduate degree, or at least to have done some programming during their undergraduate degree. The expectation is for them to be able to understand programming concepts and be able to apply these to solve problems. This course is also offered to MSc Data Analytics (DA) students who are able to take some of our CS courses in Semester 2; these students have a Maths background and have no or lower experience in programming than the other groups.

ProgSD is a highly practice-focused Bootcamp-style course that aims to standardise students foundation of most of our master’s courses. Bootcamp courses are increasingly being used by universities (Tu et al., 2018) because they can help develop students’ programming, interpersonal and communication skills. They are also being used in a variety of areas. For example, in Thalluri (2016), the intensive short program was aimed at students with little or no background in biology, chemistry or physics. Bootcamps are used to strengthen student understanding of the fundamentals of the chosen topics. A typical Bootcamp in universities lasts 12 to 16 weeks or longer (Wilson, 2018) and focuses on one programming language. However, to better serve our students, ProgSD, which is embedded in the curriculum, runs for six weeks and focuses on several subjects, specifically where lecturers are having trouble. These subjects are Java, Python, Database, Data manipulation, Data Visualisation and Unix.

In brief, our Bootcamp is a mandatory full-time 20 credits learner-centric intensive short course delivered at the start of the students’ Masters programme and designed to provide a standardised foundation to the curriculum through mainly practical tasks. Table 1 below presents a snapshot of a typical ProgSD weekly class contact timetable. The students must also take two other full-time standard-length courses (12 weeks) from week 1, which makes their timetable in the first 6 six very tight.

Table 1. Typical ProgSD Bootcamp weekly timetable.

	Monday	Wednesday	Thursday	Friday
w1	Intro lec Quiz lab	tAPP (lec 1 hour) Lab (2 hours) x 2	Project intro and group allocation Group Team project supervision (GTPS) (3–5 pm) cohort 1	GTPS (9–11 am) Cohort 2
w2-5	tAPP (lec 1 hour) Lab (2 hours) x2	tAPP (lec 1 hour) Lab (2 hours) x 2	Assessments (11 am to 1 pm) (GTPS) (3–5 pm) cohort 1	GTPS (9–11 am) Cohort 2
W6	tAPP (lec 1 h) Lab (2 hours) x 2	tAPP (lec 1 hour) Lab (2 hours) x 2	Final Lab exam (cohort 1)	Final Lab exam (cohort 2) TP submission

ProgSD aims to increase postgraduates’ practical skills and their performance in the CS courses that they take in S2. We were unable to directly evaluate student performance in those S2 courses because of the Covid19 pandemic, which led to the exam format being changed. Instead, the focus of this paper will be on evaluating students’ practical skills before and after ProgSD, as well as their perceptions of the teaching approach. Both of these issues are equally important for this course: in Computing Science, particularly in introductory programming courses, there have been reports of higher attrition rates and low pass rates due mainly to poorly designed courses, the lack of practice, timely feedback, and the difficulty in learning programming (Beaubouef & Mason, 2005; Watson & Li, 2014). One of the suggested solutions to this problem is using a pedagogical approach that promotes active student engagement and collaboration through the use of evidence-based collaborative learning and teaching strategy such as Team-Based Learning (TBL) (Walker, 2017). Furthermore, fostering collaboration from week one is very important as studying abroad for the first time can be very challenging to our mainly overseas student cohort. There is a need to foster team interaction and engagement from an early stage and to increase students’ confidence. Literature shows a growing call for using Team-Based Learning as an effective collaborative learning methodology in higher education (Vlachopoulos et al., 2020). It is more suitable to implement TBL on this course because Bootcamps, which tend to be learner-centric, optimise TBL student-centred approaches. Other studies have also combined Bootcamp and TBL. For example, Mastel-Smith et al. (2019) used it in a 16-h team-based learning Dementia Care Bootcamp. They found that the Bootcamp had a significant positive effect on the participants’ attitudes, knowledge and confidence for dementia care. Lastly, the main reason for choosing modified TBL as a method of instructions for ProgSD is because, as already explained, there are far too many topics per each lecture slot (tAPP time – 1 hour per session) which would be impossible to cover using a traditional lecture approach for the same duration. There is no extra available hour in the student weekly timetable to allow more time on tAPPs because they are also taking another two classes.

The following section will give an overview of modified Team-Based Learning and research questions. Section 3 describes the materials and methods, including details on the implementation of a modified TBL in ProgSD. Section 4 presents the results, while Section 5 presents a discussion. The paper concludes in Section 6.

Modified team-based learning

There is an increased use of modified TBL. In the literature, instructors have had to choose and practice one or more of the TBL phases or other modifications (Burgess et al., 2018; Elnagar & Ali, 2013; Inuwa et al., 2012) selectively. For instance, in an 8-week Cardiovascular Systems Block, Burgess et al. ran two iterations of modified TBL for 1.5 hours each. Their version of modified TBL included 1) compulsory pre-reading material; 2) Individual Readiness Assurance Test (iRAT) at the beginning of each class and Team Readiness Assurance Test (tRAT) in the form of multiple-choice questions; 3) Immediate feedback and 4) team problem-solving activities. However, they did not include peer evaluation. Elnagar and Ali adopted a TBL version called Lectures and Team-Based Learning (LTBL) on a 16-week Introduction to Information Technology course. Each class mixed a normal lecture with TBL activities. It would start with the instructor’s description of the subject, groups discussion, student assessment using a quiz and the provision of immediate feedback. They also used the peer evaluation component of TBL, albeit just once or twice during the semester. Inuwa et al.’s format included 1) pre-reading, 2) in-class iRAT, 3) group readiness assurance test (gRAT); 4) immediate feedback and 5) wrap-up sessions. They did not use application exercises, and a peer evaluation was performed at the end of the 11-week course.

It has been suggested that when modifying the traditional TBL structure, one must consider the ultimate value of the TBL aspects they would like to retain. For instance, Leupen (2020), who investigated TBL evidence in STEM and Health Science, stated, “it is a poor choice to use only RATs and not team applications” as it is the applications that are at the centre of TBL. However, without any prior preparation, application, or problem-solving activities will fail.

Research questions

A modified Team-Based Learning was chosen as a method of instruction. The purpose of the study is to explore master’s student perceptions of the modified team-based learning instructional approach in a Bootcamp-style course and the extent to which the Bootcamp course increases their practical experience skills. For this study, students were asked to rate the different modified TBL activities, teamwork, and their practical experience levels. This study was conducted with the approval of the University’s research ethics committee. Participants received a document containing all the necessary information about the study and their participation. All students who participated did so voluntarily, and it was made clear that the lack of participation would not affect them in any way.

The following questions frame the study:

• To what extent does the ProgSD Bootcamp course increase student practical experience in the taught topics?

• Is there any significant difference in the level of practical experience at the end of the course and across the different master’s student groups?

• Is there any significant difference in the perceptions of modified TBL activities and teamwork scores across the different master’s student groups?

• Is there any significant difference in the perceptions of modified TBL activities and teamwork scores at the end of the ProgSD course and after Semester 2 courses?

Materials and methods

ProgSD course design

Students attend two one-hour lectures and two 2-hours labs per week. They also have a 2-hour in class (lab) session per week reserved for team projects supervision (see Table 1), with tutors available. The grades of the course are divided into three different evaluations. Students have three assessed exercises based on Python and Java (weeks 2, 3 and 5) and one coursework on Linux (given on week 5 and due on week 6) (30%), a practical lab exam (Python, Java) (50%) on week 6, and a team project (20%) which is distributed at the start and submitted on week 7 to alleviate the pressure on students during that last week.

Implementation of modified TBL for ProgSD

ProgSD’s implementation of TBL consists of team formation, readiness assurance process (RAP), module application activities, individual programming tasks and team and individual exams. Teams were formed during the first week by the course. Each team consisted of 5 to 6 students grouped based on their ability. On day one, students were asked to take a quiz based on two or three questions per topic covered in the course. The quiz scores were organised in order, and the teams were created by chunking the list into teams of 5 or 6 students. The goal was to put students with similar capabilities together.

As required in TBL, the students needed to read the material before attending the classes. The modified TBL teaching approach was explained on the Moodle page, which was accessible to enrolled students well before they attended their first class. The message was reiterated at the beginning of the first class. Teaching materials were made available on Moodle at least two weeks before each class and were mainly in the form of PowerPoints. Minimum availability of two weeks was deemed necessary because of the large amounts of materials per subject that needed to be covered for each class. To maximise the benefits of the readiness assurance stage and due to the limited time, students were asked to do their iRAT out of class. Carbrey et al. (2015), for example, explored whether the iRAT could be completed at home instead of in class. There was no significant difference, and they concluded that iRAT could be done at home for more simplicity. Kirkpatrick (2017) also recommended using pre-class online quizzes for iRAT. In this study, we used the class time for team application exercises (tAPP), and we did not include tRAT due to the lack of time as tasks related to programming require more time (Lasserre, 2009), and the programme timetable could not fit additional time.

The course requires students to change their learning habits by spending a substantial amount of time in prior preparation and assessments. We use class time to reinforce the knowledge that students have acquired during prior preparation. It is very practice focussed; for example, during team application exercises (tAPP – in lecture room), teams get to a) debug codes, b) hand-execute codes, c) solve Parson puzzles (Parsons & Haden, 2006), d) write code (to solve a problem), and e) do basic Numpy maths. These exercises are done using paper and pencil only; no computers are allowed. Hands-on practice on computers only takes place in the lab sessions. The idea of using just a pencil and paper to solve these problems comes from other studies that highlight the lack of students’ sound conceptual understanding in programming (Cutts et al., 2019; Sudol-DeLyser et al., 2012).

The tAPP tasks were an application of what the students had learned. They were created based on the pre-reading material (PowerPoint slides) and the outcome of iRATs. Students were asked to solve the same significant problems and report them simultaneously. A typical tAPP contained the following instructions: Using a pencil and paper, work on problem 1 on your own for 5 minutes. Then discuss with your team members and present one solution as a team (5 minutes). Swap your solutions with another team for peer evaluation (10 minutes). The length of time spent on working alone and in teams was dependent on the number of problems. After making sure each team gave their solution to another team, the correct answer was then displayed on the screen. After marking or commenting on them, the solutions were returned to their corresponding team, and typically, students sought further clarifications from the lecturer and sometimes discussed with the other team why their answer was marked in a particular way or if they could elaborate on the comments. In the lab session, although students are supposed to work alone on their practical programming exercises, team members chose to sit together and help each other. Application-focused team programming assignments or team projects were used to foster social interaction where team members would engage in supportive communication practices, including asking for help, sharing information and conflict mediation (Rankin et al., 2007).

Participants

ProgSD ran during the first six weeks of Semester 1 (from September 2019). Participants (n = 185) were master’s students enrolled in the Programming and System Development course. The attendance rate was very good, with 99% attendance during team project activities and 97% during tAPP except during the last week (64.8%).

Data collection and analysis

Data were collected in three stages. At the beginning of the course, participants (n = 185) were asked to rate their practical experience levels on the taught topics. At the end of the six-week course and in Semester 2, during the Covid-19 lockdown, they were surveyed on their practical experience level, perceptions of the modified TBL activities and their perception of working in teams. The reason for the last survey was to find out if they would maintain the same attitudes and perceptions of ProgSD and whether it had helped them prepare for Semester 2 courses. Internal consistency of the perceptions of the modified TBL activities and perceptions of teamwork was high (Cronbach’s alpha = 0.92 for TBL and 0.93 for teamwork). The questions, including those adapted from previous research (Vasan et al., 2009), were mostly scales that included: a) perception of the modified TBL activities, b) perception of teamwork, and c) practical experience on the six taught subjects. Likert questions were rated on a scale of 1 to 10 (1 = Very inexperienced, 10 = Very experienced). The questionnaire included two open-ended questions on what they liked and disliked the most about the novel approach to teaching.

Quantitative data analysis was performed using the R Statistical Package, and data treated anonymously. Quantitative data collected was analysed using descriptive and inferential statistics. These include the Wilcoxon test, Mann-Whitney U Test and Kruskal-Wallis tests, non-parametric tests that are useful to compare groups (Pallant, 2020)

Qualitative data from students’ comments were analysed using Thematic Analysis (TA). Thematic analysis (TA), one of the most commonly used qualitative data analysis approach in Psychology (Braun & Clarke, 2006), is increasingly being used in education (Xu & Zammit, 2020). It was chosen as the approach for analysing the qualitative data because it is a theoretically flexible method (Braun & Clarke, 2019) that is independent from any particular epistemological and ontological base (Terry et al., 2017). It is also flexible in the method of data collection, sample size and analysis. Following Braun and Clarke’s six-phase analytic process, step 1 involved the familiarisation through “repeated engagement with the data”, through reading the comments several times; step 2, the “ iterative and flexible” initial code generation process of the entire dataset; step 3, searching for themes through interpretive analysis of these initial codes; step 4, reviewing themes before (step 5) defining and naming them. The last phase was about writing up. Illustrative quotations from the key themes were used in the text.

Results

Of the 185 eligible participants, 181 (98%) completed the first questionnaire, and 150 (81%) completed the post questionnaire after the ProgSD course, but only 43 (23.2%) completed the last questionnaire mainly due to Covid19 disruption. Respondents of the second questionnaire were 61 female (40.9%), 83 males (55.7%), 5 students who preferred not to say (3.4%) and 1 missing data. Ordinal Likert items were summated to produce the total scores for each Likert scale. Figure 1 presents the histograms and normal distributions of the Likert scales. Ninety-one students answered the open-ended questions in the second questionnaire on what they liked and disliked the most about the new teaching method.

Figure 1. Histograms and normal distribution of Likert scales’ scores.

Descriptive analysis of individual scale items

On average, the median score of student perceptions of teamwork was positive. All items had a median of 6 or above (Table 2). The item with a combined highest percentage ratings over 5 (from a scale of 1 to 10) and the high median was “The ability to collaborate with other students is necessary if I want to be successful as a student” (Md = 8, 82%) with CS students scoring the most. The lowest rated item for the teamwork scale was “Discussing and solving problems with my team members in the lecture room provides a deeper understanding of the topic”. (Md = 6.00, 54%). Data Analytics students’ median scores were the lowest for all three lowest items across all master’s student groups, with Md = 5 for each of these items.

Table 2. Perceptions of teamwork scale ratings for all students (survey 2).

Perception of Teamwork	n	Md	% ratings > 5
I have a positive attitude about working with my team members.	149	8.00	80%
The ability to collaborate with other students is necessary if I want to be successful as a student.	150	8.00	82%
My team works well together.	150	8.00	71.3%
There was mutual respect for other teammates’ viewpoints during TBL.	149	8.00	80.7%
Working with my team members has helped develop my communication skills.	150	7.00	76%
Working with my team members has helped me become more confident.	149	7.00	63.3%
Working with my team members has increased my interaction with other students in the classroom.	150	7.00	79.3%
I contributed meaningfully to the TBL sessions.	150	7.00	76.7%
Most students were attentive during the TBL sessions.	150	7.00	64.7%
I paid attention most of the time during the TBL sessions.	150	7.00	74%
Discussing and solving problems with my team members in the lecture room is an effective way to practice what I have learned.	150	6.00	55.3%
Discussing and solving problems with Bootcamp team members in the lecture room is a valuable use of time.	148	6.00	58.7%
Discussing and solving problems with my team members in the lecture room provides a deeper understanding of the topic.	150	6.00	54%
Discussing and solving problems with my team members in the lecture room is an enjoyable way to be taught.	150	6.00	60%

Of the 15 items that formed the modified TBL activities scale (Table 3), all but one had a median of 6 or above. The items with the same combined highest median score and percentage of ratings above 5 (Md = 8.00; 83%) were “Individual readiness assurance tests (iRAT) – (quizzes before the lectures) were useful learning activities” and “Taking individual readiness assurance tests (iRAT) – (quizzes before the lectures) is an effective way to practice what I have learned”. However, these were scored lower for DA students. This scale’s item with the lowest median was “TBL helped me increase my understanding of the course material” (Md = 5.5, 50%), closely followed by “TBL helped me prepare for the assessments” (Md = 6, 52%). DA students overall lowest median score was for the following statement “I feel more engaged in TBL activities than I usually am in traditional lecture presentations”. (Md = 5). After S2 courses, most students agreed with the statement “Programming and Systems Development has helped me prepare for my semester two courses”, with 68.2% scores of 6 and above and an average of 8.03 over 10.

Table 3. Perception of modified TBL activities scale ratings for all students (survey 3).

Perception of TBL activities	n	Md	% ratings > 5
I completed 100% of the required pre-reading material before each lecture.	150	8.00	79%
Individual readiness assurance tests (iRAT) – (quizzes before the lectures) were useful learning activities.	150	8.00	83%
pre-reading the teaching material is an effective way to learn.	149	8.00	79%
Taking individual readiness assurance tests (IRAT) – (quizzes before the lectures) is an effective way to practice what I have learned.	149	8.00	83%
I generally felt prepared for the iRAT (quizzes before the lectures).	150	7.00	77%
I learn better from lecture presentations than in a small group (like solving problems in teams in the lecture)	150	7.00	68%
TBL has increased my independent learning.	150	7.00	68%
I generally felt prepared for the Assessed exercises.	148	7.00	67%
Solving problems in teams is an effective way to learn.	150	6.00	57%
I learned useful additional information during the TBL sessions.	149	6.00	53%
TBL helped prepare for the assessments.	150	6.00	52%
Discussing with my team members during the team Application Exercises allowed me to correct my mistakes and improve understanding of the concepts.	150	6.00	57%
I feel more confident about the modules in semester two.	149	6.00	66%
I feel more engaged in TBL activities than I usually am in traditional lecture presentations.	149	6.00	56%
TBL helped me increase my understanding of the course material.	150	5.50	50%

Student perception of modified TBL

To answer the research question “Is there a difference in the perception of TBL activities and teamwork scores across the master’s student groups?” Kruskal-Wallis Tests were performed and indicated no statistically significant difference in the modified TBL activities scores between the MSc groups (n = 150, p = 0.62) and the teamwork scores between the MSc groups (n = 150, p = 0.19). Table 4 presents a descriptive analysis of the perceptions of teamwork and TBL activities across all postgraduate groups. Wilcoxon tests with continuity correction test showed that were also no significant differences in the modified TBL Activities (p = .177) scores and teamwork (p = .187) scores at the end of the ProgSD course and after S2 courses. The descriptive analysis of students’ perceptions of modified TBL Activities and teamwork at the end of ProgSD and after S2 courses are in Table 6.

Table 4. Descriptive analysis of perception of teamwork and TBL activities across all postgraduate groups.

	Teamwork		TBL Activities
MSc group	n	Median	n Median
Computing Science	27	117	27	107
Information Security	13	109	13	94
Data Analytics	42	98.5	42	98
Data Science	68	110	68	95

Students’ evaluation of their self-reported practical experience

Before ProgSD, students self-reported having from 0 months to 5 years experience in Java and Linux; 0 months to 4 years experience in Python; and 0 months to 15 years in databases, data manipulation and visualisation. Table 5 presents students’ self-reported levels of practical experience on individual subjects before and after the Bootcamp course on a scale of 1 to 10 (1 = Very inexperienced, 10 = Very experienced).

Table 5. Practical experience on individual subjects before and after course. Higher percentages for rates between 1 and 5 and lower percentages for rates between 6 and 10 BEFORE ProgSD indicate that many students self-reported having no or low practical experience in subjects. Lower percentages for rates between 1 and 5 and higher percentages for rates between 6 and 10 AFTER ProgSD indicate that many students self-reported having gained more practical experience.

	Python		Java		Data processing		Data Visualisation		Database		Linux
	Before n = 180	After n = 150	Before n = 180	After n = 150	Before n = 175	After n = 148	Before n = 176	After n = 148	Before n = 178	After n = 149	Before n = 171	After n = 150
1	27.4%	2%	32.4%	3.3%	28.6%	2.7%	33%	4.1%	18%	0.7%	41.5%	7.3%
2	16.8%	4%	8.4%	8.7%	10.3%	6.8%	13.1%	4.1%	7.9%	6%	11.1%	11.3%
3	16.2%	5.3%	11.7%	11.3%	18.3%	8.8%	15.9%	11.5%	10.7%	8.7%	8.2%	12.7%
4	10.6%	8%	6.7%	13.3%	7.4%	11.5%	10.8%	12.8%	10.1%	11.4%	8.2%	10%
5	7.8%	18%	10.6%	20.7%	16%	23.6%	9.7%	18.9%	16.3%	18.1%	9.4%	16.7%
6	6.7%	19.3%	8.4%	12.7%	8.6%	16.9%	5.7%	18.9%	14.6%	12.8%	6.4%	13.3%
7	5.6%	13.3%	8.4%	9.3%	6.3%	12.2%	8.5%	16.9%	7.9%	20.1%	7.6%	10%
8	6.1%	18%	8.4%	8.7%	3.4%	13.5%	2.8%	9.5%	7.9%	10.7%	4.7%	10.7%
9	2.2%	8.7%	3.9%	8.7%	0.6%	2.7%	0%	2.%	5.1%	6%	2.9%	6%
10	0.6%	3.3%	1.1%	3.3%	0.6%	1.4%	0.6%	1.4%	1.7%	5.4%	0%	2%

A Wilcoxon test with continuity correction showed that students’ self-reported practical experience level after ProgSD (n = 148, Md = 32) was significantly different from their self-reported practical experience before they started the course (n = 170, Md = 21.5) at p < 0.001 (Figure 2). Students’ self-reported practical experience scores in Semester 2 were not significantly different from those at the end of ProgSD in Semester 1 with p = .282. Table 6 presents the self-reported practical experience level at the end of ProgSD and after S2 courses.

Table 6. Descriptive analysis of the perceptions of modified TBL Activities and teamwork and the self-reported practical experience level at the end of ProgSD and after S2 courses.

	After ProgSD			After S2 Courses
	n	Md		n	Md
TBL Activities	150	98		43	86
Teamwork	143	96		43	91
Experience	148	32		43	35

Figure 2. Student self-reported practical experience level before and after the ProgSD Bootcamp course and during the Covid19 period.

A Kruskal-Wallis test between master’s groups revealed a statistically significant difference in self-reported experience levels across the groups with p < 0.001 (Figure 3). The effect size, calculated using eta squared formula eta2[H] = (H – k + 1)/(n – k) (where H = value obtained in the Kruskal-Wallis test; k = number of groups; n = total number of observations) was large at 0.16. A post-hoc test used Dunn (1964) Kruskal-Wallis multiple comparison p-values adjusted with the Bonferroni method. The multiple pairwise comparisons indicated that the median scores for “Data Analytics and Computing Science” and “Data Science and Data Analytics” are significantly different, with p < 0.001. Table 7 presents the descriptive analysis of postgraduate groups self-reported practical experience levels after the course.

Figure 3. Student self-reported practical experience level after the ProgSD Bootcamp course across master’s groups.

Table 7. Descriptive analysis of postgraduate groups self-reported practical experience levels after the course.

MSc group	N	Median
Computing Science	26	42
Cyber & Information Security	12	31.5
Data Analytics	42	26.5
Data Science	68	35.5

What students like/dislike the most about the modified team-based learning approach to teaching

The open-ended data analysis generated some themes suggesting that the modified team-based learning approach to teaching 1) fostered collaborative learning with positive team dynamics, 2) improved student learning through social interaction, 3) kept the students motivated and engaged; however 4) there was some resistance to new teaching methods.

Collaborative learning and team dynamics

One of the components of team-based learning is students being put in teams from day one to foster collaboration. For many respondents, the collaborative aspect, which helped them build teams skills through discussions and communication with their peers during problem-solving tasks and the team project, was widely reported. The following comments summarise the general feeling: “That’s really a good way to improve ability of cooperating with others, and learn from others while solving problem together”. (student 18); another student noted, “This helped me develop teamwork and communication skills” (student 49). The students liked that “teams were assigned in such a way that the level of knowledge among the team members was calibrated” (student 22).

However, this positive feeling of working with others was not felt by a small number of students highlighting the importance of positive team dynamics in collaboration. Imbalance of knowledge was the primary source of frustrations as some felt they had an unfair load due to some of their teammates’ limited programming skills, as highlighted in the following quotes:

“I don’t like the team project because we have the imbalance knowledge and cannot coordinate well” (student 74).

“Other members in my team know nothing about it [project]. So, I have to spend a lot of time doing more coding as well as teaching them what they don’t know” (student 20).

“Team members have different backgrounds; sometimes in order to finish team project in a high quality, some people have to take most of the work” (student 73).

This apparent lack of interaction from others, also due to the language barrier as one student remarked, “there were language gaps that made working in a team difficult” (student 63), impacted the amount of time they could spend on discussion during the team application exercises (tAPP). The number of exercises per tAPP activity meant that despite some students finding them easy as “it is kind of time-wasting for those easy questions” (student 17), many students could only spend a little time on each problem fuelling more comments on the limited discussion time. Furthermore, the physical setting of the modified TBL activities also affected the team dynamics during tAPP as “the layout of the classroom wasn’t very good for group work because you couldn’t discuss with the entire team at once”(student 33). It was “very inconvenient as lecture hall was always clustered and teams struggled to communicate”(student 47).

Social interaction

This theme revealed that the modified TBL encouraged social interaction as it gave students the opportunity to embrace diversity, to“Meet new students, whom I would not talk to otherwise. Lovely people that I would not know them if it was not for this team-based learning format”., “make friends”, “getting to know other people”, “got a chance to meet some classmates”, “Helped me interact with my classmates”, and “practise English” (some students’ comments). Although the impact of social environment on learning is very well recognised and placed at the very centre of learning, just being with people who are more knowledgeable is not enough. Indeed, according to Vygotsky, social interaction is important in cognitive development. Vygotsky’s theory of Zone of Proximal Development (ZPD) presents the dynamic relationship between learning and development. It reveals the learner’s potential and that a learner can achieve higher performance with peers’ help. This aspect of social interaction leads to a sense of community where students are involved in a community of practice, which Lave and Wenger (1990) defined as a community where people learn from each other. This sense of co-construction of learning was felt by many students who mentioned “knowledge exchange”, “sharing knowledge”, “learning from others”, indicating that the modified TBL instructional approach helped create shared learning experiences. It enhanced deeper cognitive skills with “Programming skills strengthened” (student 88) through “exposure to different opinions and views” (student 90). The modified TBL activities fostered student-centred learning and their interaction, leading to successful completion of activities and students acknowledging that they would have struggled without that interaction and support from team members.

It kept me motivated to work

Student engagement, a sociological and psychological concept (Kahu, 2013), can capture student behaviours, including student satisfaction. Student engagement components include the sense of belonging (team members have close and positive relationships), interaction and enthusiasm (Kahu, 2013). This theme revealed an interplay between student engagement components and student satisfaction, indicating that teaching and learning activities that foster engagement through (social) interaction can lead to higher student satisfaction and influence student participation in those activities. The students felt that the modified TBL activities kept them motivated and engaged as some students expressed, “It [modified TBL] kept me motivated to work” (student 52), “It was fun in lectures” (student 66), the course was “fun and useful” (student 33), “engaging” (student 93), “ It was easier to pay attention to every lecture because we were solving problems” (student 65) and “I won’t fall asleep” (student 53).

Resistance to change or additional skills needs?

Despite liking some aspects of the modified TBL, very few respondents did not like “That it [tAPP] replaced the actual lecture”. They felt that not having normal lectures and being asked to participate in iRAT meant they were teaching themselves. iRATs require the students to read the material before attending the class. An ungraded practice quiz is provided for each weekly topics to provide scaffolding to support students. That less positive feeling emerged mainly from DA students who felt that the subjects were challenging to read prior to any tAPP as they did not have a CS background.

Another source of concerns was the type of problem-solving exercises. Some students felt that having to do the exercises such as hand-coding, parsons puzzles or debugging, on a piece of paper was not a good value of time. This is summarised in these comments, “I don’t think having to handwrite code in lectures is as useful for learning to code than practising in labs etc”. (student 65), “Sometimes the exercises are not useful, for example, in my opinion, arranging code pieces in the right order” (student 22), “I don’t think handwriting answers is an effective way to learn a practical course such as this. It’s difficult to know if your code is correct if you can’t run it”(student 46).

Discussion

ProgSD, a Bootcamp course developed to standardise master’s students’ different skills sets, was taught using an instructional approach that was unfamiliar to many students. The use of a modified Team-Based Learning allowed many topics to be taught in a very short period, taking into consideration the limitations of the academic timetable while fostering team interaction and engagement. This study explored postgraduate students’ perceptions of the modified TBL activities and teamwork and whether the Bootcamp course could increase student practical experience in the taught topics. The study also intended to find out whether a modified TBL could bring the same benefits as traditional Team-Based Learning. Students’ comments reinforced the quantitative results.

Positive perceptions of modified TBL activities and teamwork across the different master’s student groups

Overall, most students had positive perceptions of teamwork and modified TBL activities, with no statistically significant difference across the different master’s groups. Results showed that, just like the original TBL, a modified Team-Based Learning could be an effective collaborative learning methodology in higher education (Vlachopoulos et al., 2020). In this study, individual readiness assurance tests (iRAT) (quizzes before the lectures) were useful learning activities and an effective way to practise what students have learned. In their study, Elnagar and Ali (2013) also found that individual readiness assurance test played an important role in student confidence and preparation readiness for their exam. We also concur with other literature that team-based application exercises (tAPPs) impact student learning (Coyne et al., 2018; Michaelsen et al., 2008). Nonetheless, based on our findings, it seems reasonable to assume that while a modified team-based learning approach to teaching may have a positive impact on student learning experience due to the collaborative nature of its activities such as tAPPs, a continued positive team dynamics where the skills and experiences are evenly distributed (Leupen, 2020) is crucial. Furthermore, agreeing with a study that investigated students’ perceptions of the value of social interaction and found that social interaction improved student learning by enhancing their knowledge (Hurst et al., 2013), we believe that our modified TBL activities fostered student-centred learning and their interaction with peers, which helped them learn, leading to successful completion of activities. Many students acknowledged that without that interaction and support from their team members, they would have struggled. This supports the notion that co-construction of learning is a crucial aspect of team-based learning (Swanson et al., 2019) because as students interact with each other, they consciously or subconsciously increase their learning (Gray & DiLoreto, 2016). Students’ comments also led to thinking that their positive perceptions may also have been influenced by their participation in a non-traditional lecture that encouraged active learning activities such as those in the modified TBL approach described in this study. A psychosocial construct that emerged was student motivation which played a critical role in student engagement in the modified TBL activities. Indeed, student engagement has been linked to learning and positive emotions (Jeno et al., 2017) and can mediate the effect of learner interaction on their perceived learning (Gray & DiLoreto, 2016).

Discussions during team problem-solving exercises

Some students felt that “discussing and solving problems with my team members in the lecture” was neither an effective way to practice what they have learned, did not provide a deeper understanding of the topic, nor a valuable use of time. These observations have also been made in the literature. For example, in their modified Team-Based learning approach to introduce computer engineering and problem-solving (programming) to freshmen students, Selim and Bender (2020) observed that the change in skills reported for group/peer learning was lower after the intervention. Another study reported that some students did not feel being in a team helped them learn course materials more effectively than studying alone nor improved their grades (Siah et al., 2019).

In this study, however, the lower rates for team discussion items may have been due to other reasons. During tAPP, the students had to solve two to 4 problems per one-hour class and could only spend a very short time discussing each problem. Student teamwork experience and discussion were also affected by the settings in the lecture room. Students sat in a linear format instead of being around the table. In some cases, students had to stand in order to be able to communicate with their team members. The conditions were not ideal, but there was no way to remediate these at the time. In a typical TBL environment, teams sit around the tables with screens that make the content visible regardless of where they sit. Finally, this negative experience in team discussions could be because we (instructors) did not promote students questioning one another and student accountability (Leupen et al., 2020).

Improving students’ practical skills through Bootcamp courses

At the end of the Bootcamp, students felt more confident in their upcoming Semester 2 courses after taking ProgSD. In Semester 2, most students agreed that ProgSD had helped them prepare for the S2 courses they were taking as they were able to apply the skills they gained through the Bootcamp in those Semester 2 courses. Similar observations were made in the literature that Bootcamps help improve skills and self-confidence in applying knowledge and skills important to students’ majors (Alavi et al., 2020, June; Fronza et al., 2020; Kenny et al., 2018; Lerner et al., 2018). That increase in confidence can last for at least six months (Selden et al., 2013), thus supporting the idea that the ProgSD Bootcamp had a positive effect on student knowledge and skills retention. In addition, knowledge retention was also due to the modified TBL instructional approach used in this study. Similar observations have been made in the literature. For example, Team-Based Learning has also been positively linked to knowledge retention (Ozgonul & Alimoglu, 2019).

Teaching a mixed-ability class

Compared to other master’s groups, the results from the DA students, though not negative, were the least positive. Individual item scores supported by DA students’ qualitative comments suggested their preference for a traditional lecture-style rather than the modified TBL activities and working in teams to solve problems. This agrees with other studies which have highlighted that resistance to a new teaching style is common (Clerici-Arias, 2021; Lasserre, 2009). On the other hand, teaching a mixed-ability class is challenging (Bennedsen & Caspersen, 2007; Mohamed, 2019). DA students may have felt that way because they have a Maths background and tend to have no or minimal prior practical experience in the CS subjects covered. They generally struggled with the ProgSD content much more than the others, which likely contributed to their assessment of the instructional approach used in this study. Some of the strategies used to help overcome the challenges posed in a mixed-ability class include using active learning activities and getting students to work with their peers (Mohamed, 2019). Our modified TBL approach encompassed these activities; however, these students may not have felt the full benefits because of its intensity. In addition to ProgSD, they took another two other courses during the same period, and ProgSD covers 6 subjects and six assessment elements during six weeks. Had one subject been the focus of the intervention like in a typical Bootcamp (Wilson, 2018), maybe their experience would have then been as positive as the other master’s groups. Remediation to these issues will require considering the specific background and needs of the DA cohort at the beginning of the course to accommodate them, rather than encapsulating all the students with the same set of initial assumptions (Lasserre, 2009). An example could be the provision of extra programming support.

Types of team application exercises (tAPPs)

ProgSD is a highly practice-focused course, and the team application exercises had to be adapted in order to fit this critical component of team-based learning. Some students complained about the type of tAPPs exercises which, as one student remarked, were not “as useful for learning to code than practising in labs” despite already having a dedicated 2-hour practice in labs for each tAPP session. The tAPPs exercises included hand-coding, Parson puzzles, code debugging, and problem-solving from scratch. The students had to use a pencil and paper only. To enhance computational thinking skills, an important development step in programming is code comprehension using exercises (Cutts et al., 2019), including the ones mentioned in this paper. Computational thinking is a foundational competency for problem-solving in STEM education (Grover & Pea, 2018). However, one of the issues identified in the literature of CS is the misleading aspect of goal-directed problem-solving labs, which let the novice student believe that the most important achievement in programming is to get programs working (Cutts et al., 2019). Cutts et al. found that using pencil-and-paper exercises, away from computers, novices were able to develop their skills. In our study, however, the comments were made by both experienced and novice programmers. For both groups, they may not have been familiar with these types of exercises that foster computational thinking or the notion of computational thinking itself. It is not surprising; although the notion of computational skills has existed for a long time, it is only in recent years that there has been increasing attention to infusing computational thinking into various spheres of education, including in schools (Herro et al., 2021). We recognise that our students’ comments may also reflect our failure, as instructors, to reiterate why these types of exercises were necessary for their learning, emphasising that notion of computational thinking.

Additionally, some of the experienced students’ conceptions emerging from how they learned programming could have been conflicting with how these concepts were being taught in ProgSD. We should have fostered “situational interest” (Thomas & Kirby, 2020) in these activities in order to help reconstruct or merge their knowledge. Situational interest refers to the “levels of interest elicited by characteristics of an immediate learning task”. In their study that sought to understand the factors that contribute to meaningful conceptual change, Thomas and Kirby (2020) found that situational interest is essential to help change learners’ conceptual knowledge.

Conclusions

This study presents a successful implementation of modified Team-Based Learning in a six-week multi-subject Bootcamp-style CS course (ProgSD). Overall, despite the fast pace and intensity of the ProgSD Bootcamp course, modified TBL, the instructional approach used to teach it, was perceived favourably by students, who also believed the Bootcamp course itself helped improve their technical skills. Our study demonstrated that our modified TBL strategy could produce the same benefits as a strict structured TBL. Students valued their interaction with others, the opportunity to learn from each other and gain multiple perspectives from their peers and solve problems together. Findings also showed that a well-formed team, an appropriate setting for activities and a clear explanation of the choice of exercises for tAPPs are important to maintain student engagement. Some issues related to teaching a mixed-ability class were also highlighted. We intend to consider these findings in the next implementation of modified TBL for the same course next academic year, with some adaptation of face-to-face activities for online delivery.

Limitations and future research

The main limitation is that the study involved participants from one university and one course. One possible bias is that those who chose to fill out the self-reported questionnaires may have favoured the modified TBL format and Bootcamp style course. The results may not be reflective of a wider population, and the qualitative comments may not be generalisable. However, we hope that the findings may inform future development of student skill sets beyond this context and may encourage future research that considers that affordance of using active learning methods such as modified TBL in postgraduate education. The study has shown that implementing a modified team-based learning approach can be as successful and benefit students as the traditional TBL. Further studies may consider investigating which of the TBL components is more suitable in CS education or other specific STEMs subjects. Additionally, there is not much research exploring postgraduate education, and results showed that how best to handle the diverse skills sets of incoming students is not an easy matter. Future research should help explore the extent of student skills sets and how these can be standardised to ensure their readiness for their master’s programme.

Practical implications

Computing Science courses tend to be highly practice-focused and could be unsuitable for the traditional TBL. Implementing a modified version of TBL would give practitioners the flexibility to redesign their courses and adopt TBL elements that would best suit their needs while still retaining the essence and benefits of TBL. The instructional approach chosen for the course must foster a sense of belonging by enabling student interaction, fostering active student engagement, collaboration and teamwork. Students are more engaged and motivated to complete the tasks and learn from each other when they are in a well-formed team created to nurture engagement and a sense of belonging. Using modified TBL in a Bootcamp style course might be of value to other practitioners who have to consider the challenges posed by a diversified student body and are having trouble with the following issues: students from different backgrounds with a variety of skill sets; large classes; a large volume of teaching material and lack of time to cover all the topics or tight curriculum; and the need to increase student performance. Finally, although this course was meant for master’s students with undergraduate degrees, most of our students had the same issues as students taking Introductory programming courses in Year 1, which is zero or minimal programming skills in one or more of the areas addressed. This can result in a high drop-out rate (Beaubouef & Mason, 2005; Walker, 2017; Watson & Li, 2014). Using a modified TBL approach to teaching might help improve retention (Walker, 2017) in those courses.

Our new proposed modified TBL format due to Covid-19

Given the dramatic change in practice because of covid-19, some elements of the modified TBL will change. While pre-reading material is already available online, the difficulty is the delivery of the tAPP, which requires face-to-face interaction with team members sitting together to discuss and solve problems. This emphasis on peer-to-peer learning facilitated by the teacher could be disrupted. One approach implemented by Jumat et al. (2020) could be to get students to remain on video calls and also use a chat feature if needed during the session while choosing their own platform for communicating with their teammates. This enables students to remain connected with their peers, even during class-wide discussions. Teams would answer questions on a shared document and present their answer through a video conferencing tool. Jumat et al. based that approach on Bandura’s social cognitive theory, which argues that individual consciousness can only be achieved through communicative interactions. However, they recognised that their approach was more suitable for the teaching of theoretical knowledge. Also, they had only 80 students and formed 13 teams.

Our course had 185 students last year and the projection for this academic year (January 2020) is 250 students with more than 40 teams of 5 to 6. Furthermore, there are only two 1-hour sessions per week reserved for this face-to-face classroom interaction. It means that we could consider teams working on their tAPP before attending the one-hour video conferencing session and use the live session to have them present their answers to the class. However, the main limitation is that there will be less time for discussion, which is not an issue for those running courses with a) more time in hand, b) a small number of students c) or courses with theory focused knowledge.

For our practice-intense course, we will have teams peer-review other teams’ code or tAPP before the session and use the live video conference session for presenting solutions, highlighting some of the key issues and discussion. This option is also beneficial in that it will help solve the issue of lack of time for tAPP and foster discussion. Finally, Jumat et al. suggest using an all-encompassing video conferencing that enables all TBL elements to be implemented without facing the possible disruption of having to switch from one platform to the other.

An extension of this work could investigate students’ perceptions of this new format, whether the same educational objectives as the face-to-face approach are achieved. It could also investigate the perceptions of the lecturers teaching the courses for which the Bootcamp course is preparing students.

Ethics approval

Ethical approval was obtained from the University’s Research Ethics Committee; application number: 300190051.

Availability of data and material

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Correction Statement

This article has been republished with minor changes. These changes do not impact the academic content of the article.

Additional information

Funding

No funds, grants, or other support was received