Students’ Learning Gains in Extracurricular Challenge-Based Learning Teams

ABSTRACT

This study aims to gain insights into what students learn in engineering-oriented extracurricular student teams. With these insights we can further students’ development of their professional identity and employability. The study involved conducting semi-structured interviews with twelve selected members of two student teams and analyzing the reported outcomes by means of deductive thematic analysis. The results of the analysis revealed that students acknowledged experiencing learning gains through their participation in the teams. Students reported acquiring fundamental engineering knowledge, along with personal and professional skills, interpersonal skills, insights into the innovation process, and leadership in engineering endeavours. These learning gains were facilitated by interactions within the diverse elements of the student teams’ ecosystem.

KEYWORDS:

• Extracurricular
• student teams
• challenge-based learning
• learning gains
• higher education
• engineering education

Introduction

Empirical evidence suggests that students engaged in extracurricular teams develop competencies promoted in education inspired by the Conceive, Design, Implement, Operate (CDIO) framework. These competencies include initiative, teamwork (Larson, Hansen, and Moneta 2006), interpersonal skills, communication, and self-awareness (Clark et al. 2015). Furthermore, the literature has underscored additional benefits, such as enhanced management and organisational skills (Thompson et al. 2013), social capital, social networks, self-confidence, and improved opportunities for finding employment in the marketplace (Stuart et al. 2011). Besides, facilitating students’ comprehension of acquired learning offers insights for reflection, enhances their self-awareness regarding their progress, and identifies areas of interest for further development or future career choices (Evans, Kandiko Howson, and Forsythe 2018; van Uum and Pepin 2022). At the institutional level, understanding students’ learning informs strategy development, facilitates program impact measurement, enhances student support, and provides insights for scaling up initiatives (Evans, Kandiko Howson, and Forsythe 2018; Kandiko Howson 2018). Considering the significance of extracurricular activities and recognising that prior studies have taken place in contexts different from those of extracurricular engineering-oriented student teams, this study seeks to provide new insights into students’ learning experiences within this specific context at a technical university in the Netherlands.

Theoretical framework

Qualitative insights into perceived changes in students’ learning can inform both students and institutions of higher education about the value of participating in extracurricular teams. To capture these insights, we must first define what constitutes a change in learning. Literature indicates that the idea of changes in learning is embedded in the concept of learning gain (McGrath et al. 2015; OECD 2012a; Rogaten et al. 2019; Vermunt, Sonia, and Vignoles 2018). Scholars have defined learning gains in various ways (Evans, Kandiko Howson, and Forsythe 2018; McGrath et al. 2015; van Uum and Pepin 2022; Vermunt, Sonia, and Vignoles 2018). For example, McGrath et al. (2015) characterises a learning gain as the difference between skills, competencies, and knowledge between two points in time. Similarly, Rogaten and Rienties (2021) describe a learning gain as an improvement in knowledge, skills, and personal development during the time students participate in higher education. In a simplified form, Pampaka et al. (2018) defines a learning gain as what is learnt between two points of time. In addition, Vermunt, Sonia, and Vignoles (2018, 272) defined a learning gain as ‘a student’s change in knowledge, skills, attitudes, and values that may occur during higher education across disciplines’. In this study, this definition was applied.

In addition, the concept of learning gains can vary depending on the context of application. For instance, Vermunt, Sonia, and Vignoles (2018) discuss a clear distinction: learning gains might pertain to subject-specific content knowledge or broader non-subject-specific skills, competencies, and personal development attributes. The latter is particularly valuable for analyzing the impact of student participation in extracurricular engineering-oriented teams. These teams lack predefined learning outcomes and span multiple disciplines, resulting in non-subject-specific learning experiences. As students engage in these teams, they undergo learning experiences that enhance their knowledge, skills, and attitudes across various dimensions. Thus, learning gains offer a useful framework for understanding the educational value of extracurricular experiences.

There are several methods available to assess students’ learning gains, including empirical observations, self-report questionnaires, surveys, rubrics, pre- and post-tests, and portfolios (Douglass, Thomson, and Zhao 2012; Evans, Kandiko Howson, and Forsythe 2018; McGrath et al. 2015). In particular, self-report questionnaires have been used in educational research in the context of co-curricular activities, as well as to assess attitudes, knowledge, and competences in higher education (Davis et al. 2023). Self-reports provide a reasonable approach to studying learning in the absence of appropriate direct measures (Ro and Knight 2016), relying on students’ self-perceptions of their skills (Picard et al. 2022).

Self-report questionnaires promote students’ self-reflection and attention to their learning experiences (Barron et al. 1998; Hmelo, Holton, and Kolodner 2000; Picard et al. 2022). They enhance the learning process by fostering self-monitoring habits crucial for effective self-regulation (Schmitz and Perels 2011). Additionally, qualitative methods, including self-report questionnaires, are well-suited for measuring individual learning gains and gaining detailed insights into students’ perspectives (McGrath et al. 2015).

When relying on self-reports to assess learning gains, there are some limitations to consider. For instance, Rogaten et al. (2019) highlight that students may exhibit overconfidence in their knowledge (Mathabathe and Potgieter 2014; Rogaten and Rienties 2021; Varsavsky, Matthews, and Hodgson 2014). Self-reported measures appear to compare students’ perceived learning gains against their subjective ‘feeling’ of learning or ‘feeling of knowing’ (Rogaten and Rienties 2021).

Gender differences significantly impact self-perceptions and self-efficacies. Research reveals that educational environments may influence men and women differently in terms of self-perception and self-efficacy (Pascarella and Terenzini 2005). Studies consistently demonstrate variations in how men and women assess their competencies, particularly in mathematics and STEM fields (Bandura 1986; Pajares 1996). Recent studies emphasise gender differences in self-perception related to skills (Reyes-González et al. 2022; Sobieraj and Krämer 2019). Besides, self-efficacy plays a crucial role in students’ learning engagement. It strongly influences their efforts, persistence, and willingness to seek assistance when needed (Linnenbrink and Pintrich 2003).

Coinciding with this evidence, Ro and Knight (2016) assert that both women and men, experiencing identical learning contexts, may self-report different learning gains due to differing perceptions and values associated with the knowledge and experiences they encounter. This phenomenon extends to other demographic characteristics of the student population.

In the case of this study, we used students’ individual self-reports to acquire insights into their learning gains. This approach is supported by our emphasis on their explanations of the processes that enabled them to achieve learning gains, the methods and resources that facilitated learning, and the types of learning gains they experienced. However, our focus does not extend to assessing the depth of the learning itself.

Regarding the different types of learning that students can acquire during their participation in an extracurricular student team, several existing frameworks provide descriptions of different sorts of categories of learning gains or learning outcomes that can be used to classify the information obtained from students’ self-reports (e.g. Bacigalupo et al. 2016; European Network for Accreditation of Engineering Education 2021; Meijers et al. 2005; Vermunt, Sonia, and Vignoles 2018). For this study, we selected CDIO Syllabus revision 3.0 as the guide for increasing our understanding of students’ self-reported learning gains (Malmqvist et al. 2022). This document provides complete and generalisable learning goals for undergraduate engineering education so that engineering programs can derive learning outcomes. The reasons for this choice are twofold. First, the CDIO syllabus refers to fundamental knowledge, personal and professional skills, interpersonal skills, and the innovation process, presenting detailed descriptions of learning outcomes that can be used to code what students self-report when asked about the learning they experience in extracurricular student teamwork. Second, the CDIO Syllabus 3.0 also presents detailed descriptions of learning outcomes associated with leading engineering endeavours, entrepreneurship, and research (Malmqvist et al. 2022). Both aspects are relevant to this research because of the characteristics of the extracurricular projects undertaken by student teams at TU/e and the focus of TU/e Innovation Space on promoting expertise in these areas.

Context of the study

TU/e innovation Space is the expertise center for challenge-based learning and student entrepreneurship at Eindhoven University of Technology. The center is the umbrella organisation for a student team program and supports around 700 students engaged in extracurricular engineering-oriented teams. These students challenge themselves to tackle some of the world’s complex challenges, working together with external companies and organisations. These challenges include sustainable mobility or accelerating the development of biosensors for health care. The composition of the teams is diverse, their members are students from different programs and levels of education, and their participation can be part-time (e.g. 10 hrs/week) or full-time (e.g. ∼40 hrs/week) depending on their availability and willingness. The teams shape their internal organisation according to their needs, and the technological component of their projects ranges from technological dissemination to integration of existing technology in a novel way and development of new technology. Team members are characterised by their intrinsic motivation, students are not rewarded in any form (e.g. no grades, no credits, no bonus, etc.), and their participation is voluntary. Finally, TU/e Innovation Space provides coaching, technical support, physical space, and points them to financial and legal advice.

Research questions

This study aims to capture students’ learning gains during their participation in extracurricular engineering-oriented student teams. We are particularly interested in ‘what’ and ‘how’ engineering students learn. Therefore, the study addresses the following two research questions:

RQ1: What do students participating in extracurricular teams learn while addressing engineering challenges?

RQ2: How do students participating in extracurricular teams learn while addressing engineering challenges?

Method

Participants

Two student teams, each facing unique challenges and objectives, were selected to provide insights into their members’ learning gains. Specifically, the challenges for the teams were ‘sustainable autonomous mobility for research activities’ (Team Artifact), and ‘accelerating the development of biosensors for health care’ (Team Event). In the study, the team members involved were in the final stage of their one-year commitment to the team, embodying diversity in gender, team roles, and study programs.

From these two teams, 12 students were selected for interviews: 7 from Team Artifact and 5 from Team Event. These students voluntarily participated in interviews after providing informed consent, without any incentives. The group comprised 8 men and 4 women, with diverse roles: 7 in managerial and admin roles, and 5 in technical roles.

Team Artifact is an extracurricular student team at TU/e. Its objective is to build an autonomous and sustainable rover for Antarctic scientific research. We interviewed students who participated in designing, building, and testing the first prototype. The members belong to bachelor’s and master’s students from programs such as applied physics, mechanical engineering, and electrical engineering. The team’s organisation includes roles that students described as managerial and technical. Managers act as representatives, make decisions, chair meetings, and manage resources. Technical roles involve identifying needs, finding solutions, manufacturing, modeling, and testing prototypes.

Team Event is also an extracurricular student team at TU/e. Its goal is to create an international network to accelerate the development of biosensors for healthcare. To achieve this goal, Team Event organises a bio-sensing competition that attracts student teams from different countries and continents. The team comprises bachelor’s and master’s students from various programs such as biomedical engineering, architecture, and medical science and technology. Team Event features well-defined roles, with a general manager, department managers, and collaborators. Students labeled these roles as managerial and administrative. Managerial tasks involve assisting people in collaborative goal achievement, dealing with partners, companies, planning, and administrating resources. Administrative responsibilities encompass contacting external partners, organising activities and logistics, managing social media content and communications, and handling finances.

Data collection

In this study, students’ perspectives on their learning gains were gathered through 60-minute semi-structured interviews. These interviews, conducted live in English, were chosen for their capacity to provide in-depth and nuanced insights (Eichelman, Clark, and Bodnar 2015; Immekus et al. 2005). Participants signed written consent forms, and the interviews were audio-recorded.

A protocol guided interviews to explore students’ learning gains and their perceived relevance. Sample interview questions include:

1. Tell us about what you learned so far in the student team. Walk us through the process.

2. Walk us through the learning you identified. Where do you see this as evident? Where did you need it?

3. Provide examples of how you reached this learning: Who or what was important in this learning? Which tasks were you able to do at the end of your participation but couldn’t do at the start?

Data analysis

The data were analyzed qualitatively by using thematic analysis. We coded and analyzed the transcribed interview data using ATLAS.ti software. The unit of analysis were the student quotes related to learning gains. Initially, we read the entire dataset without applying any codes to gain a comprehensive understanding. We considered learning gains when students explicitly mentioned gaining insight into their performance or mastering competences, following a methodology similar to Bakkenes, Vermunt, and Wubbels (2010).

Learning gains were initially coded deductively and classified according to the categories established in CDIO syllabus 3.0 (Malmqvist et al. 2022). The CDIO categories are: fundamental knowledge and reasoning, personal and professional skills and attributes, interpersonal skills, the innovation process, and leading engineering endeavours. The iterative coding process involved refining the code list, incorporating new codes, and adjusting the coding strategy.

Inter-rater reliability was checked using 20% of the coded quotes. The agreement between the main coder (first author) and a second coder (2nd author) was 87% which was considered very good (Huberman and Miles 1994).

Results

The results are organised into two sections, addressing what students learned (RQ1) and how they learned (RQ2).

What students learned

Case 1: team artifact

Team Artifact members who were interviewed included two students who played a managerial role and five students who played a technical role. These team members reported learning gains in all five categories from the CDIO syllabus 3.0: fundamental knowledge and reasoning, personal and professional skills and attributes, interpersonal skills, the innovation process, and leading engineering endeavours. In this section, we describe students’ self-reported learning gains within each category, with quotes included for added clarity.

1. Fundamental knowledge and reasoning

Six students reported significant learning gains in this category. Their involvement in conceiving, designing, constructing, and testing the prototype allowed them to bridge the gap between theoretical coursework and practical experience in constructing the Antarctic rover. Additionally, they found that this process expanded and deepened their disciplinary knowledge.

For example, Student F (technical role) expressed he deepened his fundamental knowledge from previous courses. He also increased his understanding of the impact of modifying some of the problem’s variables:

I had to conduct research on aerodynamics. In my courses, I studied the essentials, but I believe I am learning deeper how different elements, such as unclean or clean air, can affect performance. That is not commonly taught; instead, you are taught whether the flow is laminar or turbulent.

In a second example, Student E (technical role) said that she expanded her knowledge on technical topics, their application in the context of an engineering project, and implications for the diverse engineering components and systems involved in it:

I learned about batteries and solar panels, renewable energy generation in general. It really broadened my knowledge of the project. It taught me how to think about different problems, and you really have to think things through. You have to really dig deeper into how this one big thing will affect all the tiny intricacies of the system.

2.	Personal and professional skills

Two students indicated that they experienced learning gains in modelling and problem-solving when exploring solutions to a real-life technical problem. For instance, Student F (technical role) indicated that he experienced learning gains associated with modeling when he had to find alternatives for how to cool the battery system. This activity motivated him to learn about a new thermo-fluid design software that was not taught in his previous courses:

I created a MATLAB model to simulate the heat transfer during battery cooling. I applied in real life lots of theoretical things that I learned in the courses and also others that I investigated to make the model and the simulation. In addition, we got the new software that simulates fluid dynamics and heat transfer, which I will use in the simulation of some vehicles’ components. Now, I am learning how to use it.

Student G (technical role) learned the importance of precise problem definition. This insight emerged from an experience where they attempted to implement a solution without thoroughly analyzing the problem variables:

We were working on installing some gas springs to keep the cover of the vehicle up. We didn’t think things through well. Then something clicked in my head, we got a piece of paper, and we drew what we were going to do. We calculated how much pressure the gas should have. We thought, we’re going to go to a colder climate. So that’s going to decrease the pressure. Then we measured how far we could use that angle. After 30 or 40 minutes we made a little prototype. One hour later, they were installed, and they worked as we thought they would. Then, I learned that first let’s really define what we need to solve and then go ahead and do it.

In addition, all seven students, holding both managerial and technical roles, reported learning gains in adaptability, professional behaviour, and time and resource management. Regarding adaptability, students mentioned that they acquired the skill to adjust their communication and approach based on their target audience and objectives. They learned to tailor their pitches, presentations, and emails when reaching out to companies or consultants for support in addressing the team’s technical and financial requirements. The students also mentioned that they gained the ability to professionally engage with companies for technical advice and support. Additionally, they learned to maintain professionalism within the team to enhance internal efficiency. For instance, Student F, who held a technical role, reported the following:

The people on the board are really social and friendly, but they also manage to keep things professional. For me, it’s been tricky to strike the balance between being very formal and being more relaxed. They tend to be quite open and easygoing in their communication, and I’ve learned that you can still maintain professionalism while being yourself.

Finally, students reported that they improved their time and resource management. They learned how to plan their time better, use tools such as calendars, and handle team resources effectively. Time and resource management became important as they learned how to create a system for giving and receiving feedback effectively and ensuring that team members were in roles that suited them according to their motivations and team needs. They also learned about balancing work and personal lives. Student A (managerial role) shared a practical example of adopting a system he saw in his mentor’s color-coded calendar. This helped him organise tasks effectively.

3.	Interpersonal skills

All seven students reported learning gains associated with communication, which included oral presentation, communication strategy, and written communication. These skills were developed when performing the team’s operational tasks. For example, Student B (managerial role) reported:

Regarding business, the most important things that I learned were connected to communication. I’m talking about writing formal emails, making LinkedIn connections, and talking to people at events. Now, I make slide decks and tailor my pitches to different companies and different partners.

All seven students also reported learning gains in working in teams. This included conflict resolution, negotiation, multi-perspective collaboration, and stakeholder engagement. These learning gains were developed throughout the entire project when dealing with the teams’ internal operations and stakeholders. For instance, Student A indicated that he increased his awareness of the importance of including different perspectives when taking decisions with the team’s board. In addition, Student D (technical role) stated that she experienced learning gains connected with communication and teamwork:

I learned how to email and call companies, how to ask for help or advice from my team, and how to approach companies. Also, presenting skills included how to go to events and present ourselves in a way that’s appealing and gets people interested in supporting our project.

4.	Innovation process

Five students gained valuable insights related to enterprise and business contexts while performing tasks associated with management and finances during the project. For instance, Student B (managerial role) reported:

On the business side, I learned about finances; I am the financial manager. I have learned how to monitor the finances, and how to do taxes.

Four students mentioned that they experienced learning gains linked to conceiving technological solutions. These included understanding technical requirements and defining the functions of components and systems. Students recognised the importance of these aspects in minimising the number of iterations required to achieve the desired outcome. Additionally, they gained awareness of how site conditions impact certain design features.

For example, Student A (managerial role) recognised the significance of defining technical requirements. Specifically, he understood that the extreme environmental conditions dictated the material technical specifications for this particular challenge

Additionally, when it comes to rubber components, they tend to become somewhat brittle in cold temperatures. So, how can we prevent this issue? What size of screw threads should be used? These questions might seem a bit superficial, and we can always refer to specific data if needed. However, I like to think of it this way: even though in your future job, you may not be the one physically assembling a complex machine, understanding these concepts will undoubtedly be valuable.

In addition, five students reported learning gains connected with the design process. Specifically, these students utilised knowledge to solve technical problems associated with the team’s technical challenges in different disciplines such as mechanical engineering, electrical engineering, electronics, control, and computer science throughout the project’s execution. For instance, Student F (technical role) reported:

I started just applying classical thermodynamics, classical fluid dynamics, and concepts that you learn in class, and then just did more research into it. Then, I started discovering certain niches within what we’re doing and some knowledge that is more applicable to them. And that’s something that I’ve learned a lot from being on the team.

Finally, six students reported learning gains associated with the implementation of the design. During the rover prototype construction, students engaged in hands-on activities. These included developing and testing various systems and components related to energy capture, storage, distribution, control, as well as working on the rover’s chassis and transmission. In this context, Student A (managerial role) indicated that he learned about prototyping in general. Specifically, he remarked that this experience allowed him to understand the stages and the iterative nature of the prototyping process. Student C (technical role) indicated that she learned about the manufacturing process, specifically more about heavy metal machinery, how it works and what its technical capabilities are:

So, in my regular program of mechanical engineering, we used drills and mainly hand tools. I had never used the big machines that are in the workshop downstairs before. I also learned about soldering and electronics; I had never done that before. Now, I know more about how and why we have to use these technologies.

5.	Leading engineering endeavours

Five students reported learning gains in this category, which encompassed building the team, leading the organisation, and the initiation of the engineering work process. These learning gains were linked to their experience in managing an organisation with tangible and concrete goals, where coordinating efforts from individuals with diverse backgrounds was essential for success. For instance, Student A (managerial role) indicated that he noticed improvements in the way he organised the team, specifically when he had to organise tasks and define who would perform them. In this role, he learned the importance of aligning people’s motivations and their tasks to maintain motivation. Student B (managerial role) indicated that he had to deal with setting up relationships with external companies that provided technological advice for the construction of specific components. In this context, he reported that he learned how to make confidentiality agreements in order to protect, for instance, intellectual property. Finally, Student E (technical role) said that she became aware of the importance of providing and receiving feedback, and also of converting it into real-life actions when leading a team:

One of the things we learned from one of these talks with team members is that we should have more structure for giving feedback to them. Now, we have a framework in place for them to give feedback to us, the board. However, there’s no structure yet in place to give feedback in the other direction.

Case 2: team event

In Team Event, three students who played a managerial role and two students who played an administrative role, were interviewed. Students reported learning gains in the following categories: fundamental knowledge and reasoning, personal and professional skills, interpersonal skills, the innovation process, and leading engineering endeavours. In this section, we describe students’ learning gains, with quotes included for added clarity.

1. Fundamental knowledge and reasoning

Only one team member reported learning gains in this category. Student K (managerial role) indicated he acquired learning gains while organising the event. Specifically, defining acceptance criteria for the competition and interacting with technical advisors deepened their knowledge of biosensing and its socio-technical impacts:

I’ve certainly gained deeper insights into the subject. When I initially worked on that project, it was relatively obscure, and our understanding was quite limited. While I may not be significantly more technically educated now, I’ve undoubtedly expanded my knowledge in terms of biosensing as a health medium and its role as a technological advancement.

2.	Personal skills and attitudes

Two students indicated learning gains associated with self-awareness. Both highlighted the significance of thinking comprehensively and the ability to bring about positive changes, both personally and within an organisation. For instance, Student I (administrative role) became aware of the importance of not just focusing on developing technology, in this case building biosensors, but also understanding the patient’s needs in the process. It was a valuable reminder to think about the broader impact of one’s work. He told us:

I’ve seen the teams they have to make a biosensor, but they’re just so focused on their sensor that they don’t know the patients behind them. That’s why I think that it’s valuable; that’s what I’ve learned. So, now, your mind isn’t just on the solution; it’s also on the implications of what you do.

In a second example, Student K (managerial role) recounted how a previous team showed him that change is possible with effort, and he witnessed the actual changes made within their organisation. This experience motivated Student K to seek ways to improve his organisation and his work.

In addition, three students indicated learning gains connected with time and resource management. They realised that waiting for feedback from teammates is not always a reliable strategy, and proactively taking charge of planning and management is essential. This proactive approach not only allowed them to improve, but also gave them a deeper understanding of the dynamics involved in management. As an example, Student J (administrative role) indicated that she gained a better understanding of the relevance of planning:

For the second quarter, I didn’t really make a good plan And the result of that was that I spent the last week doing a lot of stuff because we had to organize a kickoff event. And there was no one who said to me, ‘Oh, you should have had good planning.’ Now, I’ve learned that I am going to design a plan that considers last quarter’s experience.

Three students also reported learning gains in connection with professional behaviour. They learned how to effectively engage with companies and how to assimilate best practices derived from these interactions and apply them to their team’s work. For instance, Student H (managerial role) indicated that he learned how to behave in meetings with partner companies and how to address them by email. In a second example, Student K (managerial role) described the necessity of developing a strategic approach when dealing with individuals in corporate settings:

It has a little bit to do with the way you plan your work and also has to do with the strategy that you create. Because, in a certain way, you are aware that these people are in different places, you need to create a plan to approach them. It’s not like calling everybody.

3.	Interpersonal skills

Two students reported learning gains related to working in teams. For example, Student I (administrative role) said that she learned how to deal with different opinions in the team when making decisions:

I learned it when working with the team; you see a lot of different opinions in a team. And sometimes everyone agrees, and sometimes people don’t agree. And you have to find the general conclusion. And it’s very hard. But it’s also very valuable to see that finally you reach agreement and the things are done.

In a second example, Student K (managerial role) reported that he learned that explaining the objectives to the team is a relevant action in order to keep members’ actions aligned with the team’s objectives.

In addition, four students reported learning gains associated with communication. These included how to modify their pitch to different audiences and how to structure emails to make communication more effective. First, Student H (managerial role) indicated that he realised the importance of adapting the pitch to the audience to obtain the desired results. In a second example, Student J (administrative role) reported that she learned that she could obtain a precise answer to her questions by predefining the structure of her emails:

I made emails like stories. And now I know that formatting emails makes it easier for the person who needs to respond to see what and where they have to answer. I learned that if you put questions in bullets, they say, ‘Oh, there are three questions; I need to send three answers’.

Finally, three students mentioned learning gains in connection with developing strategies to negotiate and resolve conflicts with external stakeholders and team members. For example, Student K (managerial role) reported:

Sometimes, our expectations don’t align with those of our stakeholders, and we need to express our needs clearly. This, in a way, necessitates effective negotiation – not just in the traditional sense but in terms of honing excellent communication skills. Whether it’s striking deals or addressing various aspects, I’ve certainly seen personal growth in this regard.

3.	Innovation process

One student, Student L (managerial role), expressed that he learned the relevance of gathering and considering the perspectives of the users when designing solutions. One of the characteristics that he remarked on was the usability of the solution:

If you want to solve a specific problem, the best technical solution might not be the most comprehensive considering the patients and doctors’ preferences. Therefore, the devices should be user-friendly as well. This involves extensive contact with patients, experts, doctors, and professors, which helps you develop the skill to ask specific questions. I learned in the process, because I wasn’t really familiar with that.

In addition, two students reported they learned more about operations management, specifically, about the relevance of creating a comfortable work environment to be more effective. They also learned that this is achieved through a combination of regular interactions, inquiries about individuals’ well-being, and the cultivation of personal relationships. Furthermore, Student K (managerial role) indicated that he became aware of the relevance of open communication and the development of personal connections that go beyond the purely professional. According to Student K, by cultivating these connections, team members are more likely to feel at ease when expressing their feelings and concerns, which in turn facilitates the prompt resolution of any issues that may arise.

Finally, two students indicated that they experienced learning gains connected with working in organisations. These learning gains involved learning how to address people in companies and how to behave in a professional way to be considered a valid interlocutor. For example, Student I (administrative role) indicated that he increased his knowledge of how companies’ employees work. In the second example, Student K (managerial role) indicated that he learned how to use different resources, such as emails, phone calls, and meetings, in the organisational context to achieve his work-related goals.

4.	Leading engineering endeavours

In this category, Student H (managerial role) indicated that he learned about implementation of innovation cycles in the team. He reported that this experience helped him to increase his forward-thinking:

Our mentor encourages us to cultivate an entrepreneurial or innovator mindset by implementing innovation cycles within Team Event. This approach accelerates our consideration of [the] team’s future. Various factors, such as financial challenges, have prompted us to collectively brainstorm about future possibilities. This continuous process extends to both minor and major decisions, not only for this year’s event but also in the broader context of the organization. It has significantly developed my skills related to innovation and fostering a culture of change and improvement.

Comparing learning gains

The students’ learning gains were systematic coded, counted, and categorised, utilising both the main CDIO categories (Malmqvist et al. 2022) and team affiliations. Each count represents an instance of a reported learning gain of a particular type. It is worth noting that identical learning gains could be articulated by multiple members within a specific team (see Table 1).

Table 1. Reported learning gains per team.

	Team artifact (n = 7)	Team event (n = 5)
Fundamental knowledge and reasoning	6	1
Personal and professional skills and attributes	13	11
Interpersonal skills	14	14
The innovation process	25	5
Leading engineering endeavours	5	2
Total	63	33

Table 1 reveals that both teams reported learning gains across all assessed categories. Despite variations in participant numbers, some trends emerged. Specifically, Team Artifact exhibited significantly greater reported learning gains within the innovation process category when contrasted with Team Event. Conversely, both teams demonstrated similar levels of reported learning gains in the personal and interpersonal skills categories. Furthermore, Team Artifact’s reported learning gains surpassed those of Team Event in fundamental knowledge and leading engineering endeavours. However, drawing definitive conclusions from these findings is constrained by the differing and small number of participant counts across the team.

How students learned

In order to answer our second research question, namely how do students participating in extracurricular teams learn while addressing engineering challenges?, we coded and analyzed the data obtained from the interviews, identifying and grouping the factors that impact students’ learning. Through this process, we discerned several factors influencing students’ learning. These factors primarily stem from students’ perceptions. Among these, we pinpointed specific activities and individuals that play pivotal roles in supporting and fostering students’ learning gains.

We categorised these activities into two groups: those internal to the team and those external to the team. Within the latter category, we further distinguished between activities internal to TU/e and those external to TU/e.

Regarding the internal team’s activities, these comprised peer feedback sessions, team reflections, practical learning through the execution of tasks connected to team members’ roles, and internal knowledge transfer sessions. For example, Student H (Team Event, managerial role) stated:

I believe that I developed my oral communication competence by doing. Being exposed to the students and the team and seeking feedback on how I was doing, allowed me to enhance this competence to another level.

Furthermore, Student I (Team Event, administrative role) reported:

I believe that taking the time to reflect on your team or department can be a valuable practice in identifying areas for improvement. It can be challenging to recognize on your own if something isn’t quite right in what you’re doing.

Regarding practical learning, students in technical roles reported learning gains related to hands-on experiences such as manufacturing processes, problem-solving, and technical knowledge. Meanwhile, those in managerial and administrative roles emphasised that they developed learning gains associated with organisational skills, professionalism, and leadership.

For example, Student E (Team Artifact, technical role) explained that she acquired knowledge and developed technical skills by means of practical hands on experiences, peer feedback sessions, and team reflection sessions when trying to find a solution for how to attach solar panels to the rover’s structure. These forms of learning were reported by all seven members of her team.

Lastly, both teams indicated that they organised team knowledge transfer sessions, in order to improve team members’ knowledge in particular topics such as electronics, materials, etc.

In terms of the team’s external activities within the university that facilitated students’ acquisition of learning gains, they reported participating in coaching sessions with previous year’s team members on technical and managerial aspects and attending TU/e innovation Space workshops on topics such as safety, electronics, and project management. Similarly, students mentioned that participation in prior curricular courses, coaching sessions with academic consultants, and technical advice sessions with innovation Space and TU/e prototyping center technical staff also promoted their learning gains. An instance of this was described by Student G (Team Artifact, technical role):

If you look downstairs, for example, the workshop has been occupied by our team for basically the entire year. Which is fantastic for us. We also really need it, and we don’t have another space to work in. And also, the senior engineers, who are part of the TU/e Innovation Space technical staff, do a fantastic job helping us, giving us advice, and designing together with us.

Finally, students reported activities that were not part of TU/e that promoted their learning. Among these, students indicated advising and coaching sessions with industry technical and business consultants and informal conversations with family and friends. For instance, Student B (Team Artifact, technical role) pointed out:

I think with the student team, and this is mainly being in the business part, I have had much more of a connection with industry life, thanks to business and technical consultants. Before this, I was a lot more academically inclined. So, my plan was also to perhaps work at the European Space Agency and kind of pursue that scientific part. But now I’ve learned I’m kind of much more drawn to the whole entrepreneurial side, to the whole business side, to the more money side, I would say.

The results showed a wide range of resources that played a role in students’ learning experiences, these are illustrated in Figure 1.

Discussion

The aim of this study was to explore what and how students learn during their participation in extracurricular teams to address engineering challenges. We observed that the students developed learning gains associated with all of the main CDIO categories (Malmqvist et al. 2022): fundamental knowledge and reasoning, personal and professional skills, interpersonal skills, the innovation process, and leading engineering endeavours.

Figure 1. Factors influencing students’ learning in TU/e extracurricular teams.

This aligns with the findings by Stuart et al. (2011), where they identified increased social capital, self-confidence, and expanded social networks as benefits of students’ participation in extracurricular activities. Furthermore, our findings align with literature indicating that extracurricular experiences, both in engineering and non-engineering context, foster the development of interpersonal and teamwork skills (Dominguez-Ramos et al. 2019; Gerber, Olson, and Komarek 2012; Larson, Hansen, and Moneta 2006). Furthermore, our findings are consistent with those of Clark et al. (2015), indicating that students engaged in extracurricular activities develop skills in team management, planning, analytical thinking, decision-making, and problem-solving. This is also supported by Gerber, Olson, and Komarek (2012), in the context of engineering-oriented extracurricular experiences.

Furthermore, within each of the main CDIO categories, students reported diverse learning gains that, in some cases, differed between the two teams. Our findings underscored that the learning gains reported by students in two distinct extracurricular student teams, each with unique goals and associated tasks, depend on the specific challenges and roles they undertake. This result is also consistent with the finding by Clark et al. (2015) that every activity does not equally influence the development of each skill.

For instance, Team Artifact reported learning gains in the innovation process category, such as defining the function of components and systems and applying knowledge to solve multidisciplinary technical challenges. These results are consistent with the findings of Amelink, Davis, and Watford (2019), Gerber, Olson, and Komarek (2012) and Dominguez-Ramos et al. (2019) who reported enhancements in innovation skills among students involved in projects integrating hands-on experiences. In addition, Team Artifact’s members reported application of disciplinary knowledge during the design and construction of the prototype. This finding is also in line with the results of Mariasiu and Raboca (2017) and Dominguez-Ramos et al. (2019) who reported this learning gain in the context of engineering-oriented extracurricular activities in automotive engineering students.

On the other hand, learning gains reported by Team Event within the innovation process category included an increased awareness of the relevance of users’ perspectives, a crucial element when striving to achieve their goal of accelerating the development of biosensors for health. These results align with the research by Sorici et al. (2023) and Huerta et al. (2022), who indicate that students in extracurricular engineering-oriented teams acquire skills that empower them to translate ideas into technology-based solutions.

Additionally, there was an overlap in learning gains. In the interpersonal skills category, both teams reported improvements in communication, conflict resolution, and collaboration. This was due to the challenges requiring interactions with peers and stakeholders, facilitating skill development. Similar situations occurred in personal and professional skills, with members reporting learning gains in time and resource management, and professional behaviour. Students adjusted their behaviour to meet stakeholder expectations regarding communication etiquette, meeting scheduling, and presenting information. Furthermore, both teams had to manage limited resources, including members’ time availability and project funding.

Connected to the aforementioned, the significance of the learning context within the extracurricular program in which the teams are immersed cannot be overstated. Interactions with its diverse elements, both internal and external to the team as depicted in Figure 1, facilitated the development of various types of learning gains linked with the tasks associated with each student’s role in the context of the team’s challenge. These interactions also provided access to material and economic resources, knowledge, mentoring, coaching, and real-life experiences that enhanced students’ learning process. These findings are consistent with the research of both Klaassen, Hellendoorn, and Bossen (2024) and Helker et al. (2024) in the context of challenge-based learning within the engineering curriculum. They emphasised the significant impact of interactions with peers, stakeholders, and professionals on students’ acquisition of learning gains. Specifically, Klaassen, Hellendoorn, and Bossen (2024) underscored the development of interprofessional competence and a comprehensive understanding of the interdisciplinary context resulting from such interactions.

Lastly, we observed significant educational value added by extracurricular student teams in developing skills and competences less likely to be cultivated within standard coursework and curricular activities due to constraints such as semester duration, course-specific learning outcomes, and limited contact with real stakeholders. Within these teams, members increased their skills to tailor communication for communicating with companies, expand professional networks on events and on-line platforms, and address conflict resolution and negotiation with external partners.

We should also emphasise that if students or teams wish to experience specific types of learning, aligning these learning goals with the team’s challenges and roles is crucial. This alignment maximises the chances of successfully achieving their learning goals. While some teams currently promote the definition of learning goals at the beginning of students’ participation, this practice is not common.

When a team decides to make learning a relevant goal for the team and its members, we suggest that team members define their learning goals in collaboration with the team at the beginning of their participation, regularly tracking and reflecting on their progress, and receiving feedback on their development. Co-creating approaches and placing the alignment of students and team learning needs at the center of the discussion offers benefits for both the team and the individual learners. First, it facilitates the identification of learning gaps, allowing teams to proactively seek external resources such as workshops, coaching, and experts from academia or industry. Second, this approach enhances the learning experience and provides students with opportunities to bridge knowledge and skill gaps by adding new resources to those they reported in this study: doing, peers in the team, and reflecting in team sessions. Third, when reflecting on their learning process, students are encouraged to integrate theory with practice (Malaysia Ministry of Education 2015; Wong et al. 1995). The latter is of particular interest for engineering education, because reflection allows students to develop the ability to gain and utilise knowledge from practice and integrate the knowledge acquired in their previous subjects to look for solutions in an authentic context in a meaningful way. Lastly, the aforementioned benefits enable students to be better prepared to take the lead in maintaining their competence through continuous professional development, which is aligned with EUR ING SPEC professional competences (Engineers Europe n.d.).

This approach also poses some challenges that need to be addressed by the teams. For instance, setting learning goals and analyzing progress towards them requires a reflective process that may need scaffolding. In addition, reflecting critically requires higher-order cognitive processes and metacognition, and these capacities may not be present in some students (Coulson and Harvey 2013). Furthermore, this poses a challenge to the organisation of the student teams, because carrying out this process requires the team to develop reflective skills and needs support at the beginning and during its enactment.

Contributions

Our findings contribute to engineering education research. They enhance our understanding of the educational benefits of extracurricular activities for engineering students. This helps bridge the gap between previous studies conducted in other contexts, such as the studies by Stuart et al. (2011) and Larson, Hansen, and Moneta (2006), which did not specifically focus on engineering students, the study by Clark et al. (2015), which only partially addressed engineering students, and the work of Mariasiu and Raboca (2017), focused on automotive engineering students. Besides, we identified that the findings from these studies can be transferred to the engineering education context.

This study explored the application of CDIO syllabus learning outcomes in engineering education research to analyze students’ self-reported learning gains. Our findings suggest that the categories and comprehensive descriptions provided by CDIO facilitate the process of identifying and classifying students’ learning outcomes. In addition, this study revealed that the descriptions align with students’ reported learning gains properly, and we did not find self-reported learning gains that were not covered in the CDIO syllabus. Furthermore, we observed that the syllabus is particularly useful when describing learning gains associated with the innovation process, as it provides detailed descriptions for each stage that composes this process.

Limitations and future research

One limitation of this study is its small sample size, involving only two teams. Consequently, it may not fully encompass the diverse spectrum of learning gains and experiences that students can self-report. Factors such as team culture, the complexity of challenges (both technical and societal), team dynamics, project phase, and stakeholder engagement significantly shape participants’ perceptions and learning gains. It’s crucial to recognise that various experiences and activities contribute to skill development in distinct ways (Clark et al. 2015).

Another limitation of this study is its reliance solely on self-reported data. Self-reported learning gains are influenced by various factors. Firstly, overconfidence may lead participants to inaccurately assess their progress (Rogaten et al. 2019). Secondly, inherent subjectivity arises because individuals interpret their experiences differently (Rogaten and Rienties 2021). Additionally, participants’ self-perceptions and self-efficacy can skew their evaluations of development. Moreover, gender differences in educational contexts impact how individuals report learning gains, with men and women attributing different values to their experiences (Bandura 1986; Meece and Courtney 1992; Pascarella and Terenzini 2005). These differences may also relate to cultural background and socioeconomic status (Ro and Knight 2016). Lastly, implicit, nonconscious learning remains unaccounted for, as noted by Bakkenes, Vermunt, and Wubbels (2010).

Another limitation we found is that we only considered quotes as reports of learning gains when students explicitly mentioned gaining deeper insight into their performance, mastery of knowledge, skills, or competences. However, existing literature encompasses additional learning categories where students have reported understanding how a skill functions or recognising positive shifts in their perception of the value and importance of generic skills – insights that were not apparent before their participation (van Ravenswaaij et al. 2022).

In the interviews, students self-reported their learning gains in English, despite it not being their native language. This linguistic difference may affect the depth of their descriptions, given potential limitations in vocabulary and expressing intricate ideas in a non-native tongue.

To overcome these limitations, future research might expand the sample size by including more teams and students, allowing for the use of quantitative research methods to generalise the findings. Including diverse teams with varying challenges, structures, and stakeholder engagement levels could shed light on how these variables affect students’ learning gains. Additionally, a larger participant pool could deepen our understanding of other factors, such as the influence of gender, sociocultural context, and socioeconomic status on perceived learning gains in extracurricular student team experiences.

Furthermore, by using a mixed-methods approach – combining diverse evidence collection methods like observations, surveys, guided reflections, portfolios, and self-reflection journals – we can delve into students’ learning paths and experiences. Additionally, incorporating pre-assessments and post-assessments, along with cross-referencing self-reported learning gains with feedback from peers, coaches, or advisors, would yield a holistic understanding of participants’ development.

Finally, studying the performance of student team alumni in curricular courses or other contexts, like future work, could provide valuable insights into the impact of student team participation on learning gains.

Conclusions

This study offers new insights into students’ learning gains within engineering-oriented extracurricular teams at a technical university in the Netherlands, complementing and validating the findings of previous studies on the realm of engineering education. Additionally, this work highlights the educational value of extracurricular teams, where students with various interests and backgrounds take the lead in their learning. In this educational context, students have the possibility to work across different phases of year or multi-year projects, and interacting with varied stakeholders offering real-life experiences that promote the development of diverse types of knowledge, skills, and competences required in the professional life. Finally, student teams offer the chance to be exposed to experiences that help clarify students’ preferences regarding their future career paths.

Ethical approval

This study received approval from the Eindhoven University of Technology ethical review board (ERB2022IS1) on 03-02-2022. Participants’ names were anonymised, and consent forms were signed.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work received support from TU/e innovation Space, Eindhoven University of Technology.

Notes on contributors

Eugenio Bravo

Eugenio Bravo is an engineer and educational researcher working on the “Extracurricular Learning & Competence Development” project at Eindhoven University of Technology (TU/e) Innovation Space. He has played a key role in implementing Challenge-Based Learning courses in engineering higher education. His research is centered on innovating engineering education, with a special focus on improving how we support self-directed learning within open and flexible learning environments.

Dury Bayram

Dury Bayram is an Assistant Professor at the Eindhoven School of Education (ESoE), Eindhoven University of Technology (TU/e). Her research interests are Socio-scientific Issues (SSI), science education for citizenship, pedagogy of physics, Pedagogical Content Knowledge (PCK) of physics/science teachers, formative evaluation of SSI lessons, and innovation in education.

Jan T. van der Veen

Jan T. van der Veen is a full Professor at the Eindhoven School of Education (ESoE), Eindhoven University of Technology (TU/e). His research focuses on innovating STEM education in secondary and higher education including the professional development of STEM educators. He contributes to the international community working on rewarding teaching excellence.

Isabelle Reymen

Isabelle Reymen is the Scientific Director of TU/e Innovation Space and a Professor of Design of Innovation Ecosystems in the Innovation, Technology Entrepreneurship, and Marketing (ITEM) group at Eindhoven University of Technology. She founded TU/e Innovation Space with the ambition to fundamentally change education, and after seven years, she leads an award-winning team dedicated to advancing educational innovation with a commitment to the highest standards of quality.