Students’ self-reported learning gains in higher engineering education

ABSTRACT

In higher engineering education, students’ learning outcomes are typically measured via tests and examinations. In that way, so it is claimed, students have reached the learning goals intended by the teachers. However, students’ learning experiences and perceived learning gains remain unclear. To gain insights into their perceived learning gains, thirteen students from a Dutch technical university were interviewed. This university is in the process of changing its curriculum from teacher-centered to student-centered approaches. The results show that students’ reported learning gains could be grouped into five different strands: (1) the disciplinary conceptual and procedural knowledge strand; (2) the general cognitive learning strand; (3) the affect, thought and learning strand; (4) the teamwork and communication strand; and (5) the entrepreneurial learning strand. Moreover, we nuanced these five strands with the help of the student interviews and the relevant literature. The findings of this study can be valuable and helpful for teachers and curriculum leaders for their course and curriculum design.

KEYWORDS:

• Learning gains
• higher engineering education
• self-report
• pie chart drawing
• teacher-centered and student-centered education

1. Introduction

In many countries (e.g. United States of America, Denmark, Sweden, the Netherlands), higher engineering education has developed from teacher-centered instruction to student-centered approaches, such as problem-based, design-based, and challenge-based learning (e.g. Gómez Puente 2014; Membrillo-Hernández et al. 2019). In these student-centered approaches, students collaborate and work on open-ended problems or challenges in teams.

A change from teacher-centered to student-centered education impacts at different levels of the curriculum: the intended, implemented, and attained level (Van den Akker 2003). The attained curriculum addresses learners’ experiences (the experiential curriculum) and their learning outcomes (e.g. from examinations or tests). However, the outcomes measured by tests and examinations do not necessarily correspond to students’ self-perceived learning experiences and gains, and the latter might influence students’ decisions for specific courses or subjects during their higher education, or choices regarding their future profession. Furthermore, qualitative insights into students’ perceived learning gains can inform educators who are developing their courses (e.g. from lecture-based to student-based education).

In the current study, we investigate students’ perceived and self-reported learning gains in higher engineering education at a Dutch technical university that is changing a large part of its curriculum from teacher-instructed to student-centered approaches. The aim of the study is to provide insight into students’ explanations of their perceived learning gains, to obtain a qualitative overview that informs curricula and could be the first step towards framing students’ learning gains in higher engineering education. Our research question is the following: Which learning gains do students report to perceive in higher engineering education that is changing from teacher-centered to student-centered approaches?

After this introduction, we present the theoretical frames we used in this study, followed by the methods. Subsequently, we outline and discuss the results, before drawing conclusions and discussing the limitations of the study. In addition, we provide the implications of the research for policy and practice.

2. Theoretical frames

In this section, we explain the following theoretical frames: 2.1 Learning gains in higher education; 2.2 Learning gains in higher engineering education; 2.3 Measuring learning gains; and 2.4 Teacher-centered and student-centered engineering education.

2.1 Learning gains in higher education

Large-scale studies, for example in the US and the UK, address learning gains in higher education (Rogaten et al. 2019). Pascarella and Blaich (2013), for example, described students’ cognitive and social learning gains in a large-scale US study. This attention to learning gains in higher education can be explained by a focus on the contribution to or value of higher education for students’ learning. It has been argued that, as a learning outcome is not automatically a good representation of educational quality or students’ overall learning and development, the concept of learning gains could be promising as a (first) indication of the value of higher education, although it should be measured in a consistent way (Rogaten and Rienties 2020).

‘The concept of “learning gain” is defined as the “distance traveled”, or the difference between the skills, competencies, content knowledge and personal development demonstrated by students at two points in time’ (McGrath et al. 2015, xi). When addressing students’ learning gains, the literature points to the issue that it should be clarified what these learning gains consist of. According to Vermunt, Ilie, and Vignoles (2018), learning gains comprise a change in knowledge, skills, values, and attitude. Learning gains should be distinguished from, for example, graduate attributes, competencies, skills, and capabilities as these measure results or outcomes instead of change.

Regarding the types of learning gains that are studied, research in psychology and education fields usually differentiates between three types of learning: cognitive, behavioural and affective learning (Rogaten et al. 2019; Singleton 2015). Singleton refers to this distinction as ‘head’ (cognitive area), ‘heart’ (affective area), and ‘hands’ (behavioural area), which has its origins in studies of, for example, Dewey (1910: learning from and reflecting on experience) and Bloom (1956: cognitive, psychomotor and affective domain). Cognitive learning includes, for example, knowledge and understanding, metacognition and self-regulated learning. Attitudes, interests and confidence are viewed as part of affective learning, and on-task behaviours and team work are part of behavioural learning (Rogaten et al. 2019; Singleton 2015). Therefore, when studying learning gains in higher education, cognitive, affective and behavioural factors are to be taken into account.

A framework that includes these types of learning is the framework of Vermunt, Ilie, and Vignoles (2018). They studied learning gains in higher education with the aim of developing an instrument to measure these gains. Their framework consists of four components: a cognitive; metacognitive; affective; and socio-communicative component. The cognitive component includes, among others, critical thinking and analytical thinking. Self-regulation and learning to learn are part of the metacognitive component. Examples of the affective component are motivation to learn and attitudes towards own discipline and towards learning. And the socio-communicative component includes, for example, belonging to social learning networks and communication skills. In addition to the four components, the authors distinguished between three dimensions: (1) view of knowledge and learning (e.g. view of intelligence), (2) the research dimension (e.g. attitude to sharing ideas), and (3) the moral dimension (e.g. moral reasoning). The framework of Vermunt, Ilie, and Vignoles (2018) clarifies learning gains in higher education in general, but lacks a focus on higher engineering education.

2.2 Learning gains in higher engineering education

An overview on goals and expected learning outcomes in higher engineering education which could be used to study learning gains, is provided by the CDIO initiative (Conceive, Design, Implement, Operate). This worldwide initiative focuses on contemporary engineering education and addresses knowledge, skills and attitudes that are important for the future career of the engineering student. The CDIO syllabus contains four categories of expected learning outcomes: (1) disciplinary knowledge and reasoning, such as knowledge of underlying mathematics and sciences; (2) personal and professional skills and attributes, such as analytical reasoning and problem-solving; (3) interpersonal skills: teamwork and communication, such as communications in foreign languages; and (4) conceiving, designing, implementing and operating systems in the enterprise, societal and environmental context – the innovation process (Crawley et al. 2011).

The CDIO learning outcomes connect to the ‘universal educational taxonomy developed by UNESCO’ (Crawley et al. 2011, 8). In this taxonomy, four pillars of education are explained: learning to know, learning to do, learning to be, and learning to live together (Delors 1996). The first type of learning in this taxonomy focuses on learning to know. ‘Learning to know’ refers to the knowledge that people need to work on something in depth, which is addressed in the first category of the CDIO syllabus as ‘technical knowledge and reasoning’. In addition, ‘learning to be’ includes personal abilities, such as being a responsible and autonomous person. This is addressed as ‘personal and professional skills and attributes’ in the CDIO syllabus. ‘Learning to live together’ is focused on, for example, working together with other people, and is phrased in the CDIO syllabus as ‘interpersonal skills: teamwork and communication’. Finally, ‘learning to do’ refers to skills that people need and is defined in the CDIO syllabus as ‘conceiving, designing, implementing and operating systems in the enterprise, societal and environmental context’.

The CDIO overview of goals and expected learning outcomes is similar to other categorisations of learning outcomes. For example, the graduate attribute profile of the Washington Accord (an international agreement between accreditation organisations regarding higher engineering education) contains elements, such as: engineering knowledge, problem analysis, the engineer and society, individual and teamwork, and project management and finance (IEA 2014). These elements match the programme outcomes regarding the EUR-ACE (EURopean-ACcredited Engineer) label. This label can be awarded to engineering degree programmes by accreditation agencies who are authorised by the European Network for Engineering Accreditation. Examples of expected outcomes of these programmes are: knowledge and understanding, engineering analysis, communication and team-working, and lifelong learning (ENAEE 2015). In addition, the ABET engineering accreditation commission formulated outcomes that students are expected to achieve when they have graduated from engineering education, which match the CDIO learning outcomes. Examples are: science and mathematics principles, engineering design, effective communication, professional and ethical responsibilities, and collaboration in a team (ABET 2019). This connection of the CDIO syllabus to other categories of learning outcomes and the types of learning of UNESCO might be promising to study students’ learning gains in higher engineering education, although it is not clear yet how these learning gains can be measured.

2.3 Measuring learning gains

There are several studies which describe the measurement of learning gains (e.g. Aloisi and Callaghan 2018; Arico et al. 2018). For example, Arico et al. (2018) described that student marks are routinely used in education and could be used to measure learning gains. However, a difference in marks between two subsequent courses might not only correspond to a learning gain, but can also be influenced by differences in assessments by examiners. Another option to measure learning gains is via pre-tests and post-tests. Rogaten et al. (2019) found that these tests are mostly used to measure cognitive learning gains of one group of students of a particular course, or more groups when comparing lecture-based education with modern types of education. According to Rogaten and Rienties (2020), a disadvantage of tests is that they usually measure knowledge and understanding, and lack authentic learning gains.

In addition to tests, questionnaires are often used to measure learning gains. Leandro Cruz, Saunders-Smits, and Groen (2020) reviewed competency methods in engineering education and found that self-report questionnaires and rubrics were mostly used to measure communication, teamwork, lifelong learning and innovation/creativity competencies. In addition, Purzer, Fila, and Nataraja (2016) evaluated assessment methods and found that in engineering entrepreneurship education, self-report surveys were used the most to measure students’ knowledge, skills and attitudes regarding, for example, business planning, communication, leadership and teamwork. An example of a study that used self-reports is the study of Turner et al. (2018). They asked students to assess their own learning regarding research methods, using a survey with Likert scales at the beginning of a study year, at the end of the same study year, and one year later. The results were compared to measure students’ learning gains. According to Rogaten et al. (2019), self-reported learning gains are mostly used in studies that measure affective and/or behavioural learning gains in addition to cognitive learning gains. A disadvantage of using self-reports to measure learning gains is that students can be overconfident about their learning gains (Rogaten and Rienties 2020).

In the current study, we address several types of learning gains and chose to determine students’ self-reported learning gains. According to Arico et al. (2018), ‘empirical evidence from the US shows that self-reported data from student experience surveys display good correlation with student Grade Point Averages (GPAs) and perform better than standardised tests, such as the Collegiate Learning Assessment, at mirroring student learning’ (Arico et al. 2018, 251). Although students can be overconfident about their learning gains in self-reports, our focus is more on the areas of learning gains perceived by the students and their qualitative explanations regarding their learning gains than determining their quantitative learning gains. This matches recommendations for researchers to explore other methods in addition to the mostly quantitative studies regarding learning gains in higher education (Rogaten et al. 2019). A recommendation is, for example, to conduct case study research to assess students’ level of expertise in engineering entrepreneurship (Purzer, Fila, and Nataraja 2016). In addition, Kinchin (2016) addressed learning gains in biology education and claimed that it is important to conduct qualitative case studies ‘to reveal what is happening beneath the surface as our students move between the two points in time highlighted by McGrath et al. (2015)’ (Kinchin 2016, 359). Furthermore, according to Baume (2018), it is important to consider the individual perspective of students when determining students’ learning gains. Baume explained that when two students enter higher education with the same scores on an entry test, a student who graduates without an honours degree might have gained more in learning than a student who graduates with an honours degree. Baume clarified that the student with an honours degree might have no interest in the subject that was studied and might not know how to progress after higher education, while the student without an honours degree might have created a network of contacts during higher education to get the job s/he wanted regarding the subject that was studied. Therefore, the latter student has gained, for example, knowledge about networking that the first student might not have gained. Baume suggested that self-reports might be able to take into account this individual perspective.

2.4 Teacher-centered and student-centered engineering education

Both teacher-centered and student-centered education are part of this study. We define teacher-centered education as a lecture-based approach in which the teacher explains the content to the students and students can ask questions about these instructions and content (e.g. Martin, Rivale, and Diller 2007). In contrast, in student-centered approaches, students collaborate in teams to solve problems. An example of a student-centered approach is problem-based learning (PBL), which includes collaboration between students in small groups, a tutor who facilitates learning, authentic problems, finding and using information connected to the problem, and students who direct their own learning (Dochy et al. 2003). When comparing problem-based and traditional learning, Vernon and Blake (1993) found in their meta-analysis that students in PBL focus more on self-directed learning than students in traditional education. In addition, Dochy et al. (2003) found that students in PBL acquire more skills than in traditional education. The results on knowledge acquisition in favouring traditional learning or PBL are less clear, although the meta-analysis of Gijbels et al. (2005) revealed positive effects of PBL on students’ ability to understand connections between concepts.

In the current study, similar student-centered approaches are centralised: design-based learning (DBL) and challenge-based learning (CBL), as these approaches were part of the teaching and learning approaches at the technical university where this study was conducted. According to Gómez Puente, Van Eijck, and Jochems (2011), DBL is similar to PBL, but centralises ‘the design of artefacts, systems and solutions’ (137). Elements that contribute to a good design are, for example, exploration of the problem, using an iterative design method, and exploring alternative solutions (Mehalik and Schunn 2006). Key characteristics of DBL include ‘open ended, hands-on, authentic and multidisciplinary design tasks resembling the community of engineering professionals’ (Gómez Puente, Van Eijck, and Jochems 2013, 717). During DBL, students work together and plan activities, make predictions, and test and communicate while creating and improving their solution for a design problem (e.g. Gómez Puente, Van Eijck, and Jochems 2013, 2014). Design elements that are frequently part of DBL are exploration of the problem and graphic representations, building a normative model, and validation of assumptions and constraints (Gómez Puente, Van Eijck, and Jochems 2011). In the study of Gómez Puente, Van Eijck, and Jochems (2014), DBL contributed, for example, to students’ awareness of the process of solving design problems.

An approach similar to DBL, is CBL in which a challenge is central. During CBL, students collaborate in (multidisciplinary or discipline specific) teams and with stakeholders on authentic and open-ended challenges (e.g. Gallagher and Savage 2020; Membrillo-Hernández et al. 2019). Mostly used keywords regarding CBL are, for example: solve challenges, real-world problems, collaborative, and ask questions (Leijon et al. 2021, 4). In addition, key features of CBL include global themes, real-world challenges, collaboration, multidiscipline and discipline specific approaches, and innovation and creativity (Gallagher and Savage 2020, 9). CBL contributes, for example, to innovative thinking (Martin, Rivale, and Diller 2007) and integrating and synthesising concepts (O’Mahony et al. 2012).

The studies mentioned, focused on specific learning gains regarding DBL and CBL. In the current study, our aim is to take the first step in developing an overview of students’ (perceived) learning gains in higher engineering education that includes CBL and DBL. As these student-centered approaches reveal promising results and are upcoming in higher engineering education, whilst teacher-based education is still a common practice at the same time, both student-centered and teacher-based education approaches are part of the current study.

3. Method

3.1. Context of the study

This study has been conducted at one of the technical universities in the Netherlands. At this university, different educational programmes can be chosen regarding sciences (e.g. Applied Mathematics, Applied Physics), core engineering studies (e.g. Automotive Technology, Mechanical Engineering) and social engineering studies (e.g. Psychology and Technology, Sustainable Innovation). Engineering education at this technical university is in the process of changing from teacher-based to student-centered education. DBL has been introduced and implemented in education at this university for more than 20 years now (Gómez Puente, Van Eijck, and Jochems 2011). More recently, CBL has been heralded as one of the university’s main education strategies towards 2030. At the moment when the study took place, some courses still focused on traditional instruction by the teacher. Others already had been developed into courses in which students design solutions for problems or are challenged to solve problems in collaboration with clients from industry and companies.

3.2. Participants

The perceived learning gains of students in both teacher-centered and student-centered education are centralised in the current study. Therefore, we selected students for our study who participated in both student-centered and lecture-based courses. We interviewed 13 students in different fields of higher engineering education at the technical university. We chose this relatively small number of participants to gain an in-depth perspective on explanations of their learning gains. We decided to use challenge-based (CB) courses to recruit students for our study as these students would have experienced both teacher-centered and student-centered education. At an information market of a CB course, we asked students to participate in our study. We chose this event, because there would be students present from the sciences, social engineering and core engineering studies. The recruitment of students during this event resulted in nine participants for our study. Since these participants consisted mostly of science students and social engineering students, we approached students from another CB course focused on core engineering. Four additional students were willing to participate in our study. Informed written consent was obtained from all participants. In Table 1, an overview of the background of the students is provided. This information was provided by the students themselves prior to their interviews.

Table 1. Background information of the students.

Variable	Category	N
Gender	Man	5
	Woman	8
Age	18	1
	19	4
	20	6
	21	1
	22	1
Nationality	Netherlands	12
	Other (Albanian)	1
Year of study	Second year of Bachelor’s degree programme.	11
	Third year of Bachelor’s degree programme.	1
	Fourth year of Bachelor’s degree programme.	1
Study	Sustainable Innovation	4
	Applied Mathematics	3
	Industrial Design	3
	Mechanical Engineering	2
	Computer Science and Engineering	1

Table 1 clarifies that we interviewed more women than men. Our results showed that there were no differences between genders regarding their explanations of learning gains in the interview data. Most interviewed students were 19 or 20 years old, had the Dutch nationality and were in the second year of their Bachelor’s degree programme. Two students were further along with their studies, but were included in the current study as they were part of the CB courses in which we recruited participants. We incorporated their results as part of the current study, because their data were similar to the data of the other students. Students from different fields of engineering education participated in our study: from the science studies (Applied Mathematics), the core engineering studies (Computer Science and Engineering, Mechanical Engineering), and the social engineering studies (Sustainable Innovation, Industrial Design).

3.3. Instrument and procedure

We used semi-structured interviews in which we determined students’ own perceived learning gains during their time in higher engineering education. We interviewed students mostly in pairs to enable them to build on each other’s ideas when thinking about their perceived learning gains. Due to different schedules of the students, ten students were interviewed in pairs and three students were interviewed individually. The responses of the individually interviewed students were comparable with the students who were interviewed in pairs.

We explained to the students that the interview entailed their learning gains during their time at the university (from the start of their university career until the moment they were interviewed about their learning gains). Subsequently, we asked participants what they perceived to have learned during the time they entered the university until the present. In addition, the students were asked to describe and explain each learning gain they mentioned, and to include examples of the learning gains and courses or moments in which these gains had occurred.

After the students had described their learning gains, we presented them with the strategy ‘pie chart drawing’ (PCD). Pie charts are used, mostly in quantitative studies, ‘to compare the parts of a whole’ (Sadiku et al. 2016, 12). For example, in the study of Bieńkowska and Brol (2013), pie charts were used to visualise the actual and desired competences of an employe, with each part of the pie chart representing a different competence. In the current study, the PCD enabled students to visualise and provide an overview of their perceived learning gains. To facilitate the PCD, we provided the students with a piece of paper with a circle on it, and asked them to divide the circle into different parts, each part representing a learning gain that they had experienced. Furthermore, we asked the students to write a brief explanation for each learning gain, including the course or moment in which the learning gain was perceived. The students made the pie chart without our input and based it on their descriptions of learning gains which they had mentioned during the interviews.

3.4. Data analyses

To analyse the qualitative data, we used a combination of inductive content analysis (the PCD data) and deductive content analysis (the interview data) (Elo and Kyngäs 2008). We labelled the remaining interview data regarding students’ learning gains as ‘other’. Within that category an inductive content analysis was applied. Below, the analyses are explained in order to comprehend the steps that were taken.

First, we analysed each PCD consisting of a pie chart divided into different parts that represented the students’ self-reported learning gains perceived in their courses at the university. The PCD data provided an overview on each participant’s learning gains and were analysed inductively. The data of the participants were placed in a table in order to compose a data display of all perceived learning gains (Miles and Huberman 1994). Each learning gain was placed on a different row. For each row, in the first column, the name of the participant was written and in subsequent columns, the name of their study, their learning gain, a written explanation of their learning gain when they had provided one, and the course(s) or moment(s) in which the learning gain was perceived. Subsequently, we grouped comparable learning gains together that formed categories within the data. For example, learning gains that various students mentioned as ‘theory’ or ‘theoretical knowledge’ were placed together. After grouping the learning gains, certain subcategories emerged. In the previous example, we named the subcategory ‘theoretical knowledge’. In addition, we observed whether subcategories could be placed into larger categories. For example, the subcategories ‘theoretical knowledge’ and ‘applying theory in models, graphs and programmes’ were placed into the category ‘disciplinary conceptual and procedural knowledge’.

Secondly, the interview data were transcribed and divided into meaningful fragments that included (an explanation of) a learning gain. Each fragment was labelled deductively according to the categories and subcategories that had emerged from the PCD data. As the PCD data represented an overview of students’ learning gains and the interviews contained their elaborate explanations, most of the interview data could be labelled with the formulated categories and subcategories. When a student mentioned (an explanation of) a learning gain that could not be labelled with the composed categories and subcategories, we labelled these as ‘other’. We compared the learning gains in the category ‘other’ and observed inductively whether there were similarities between the comments that would be informative for the current study.

4. Results

In this section, we describe the results on students’ perceived and self-reported learning gains by presenting each category and subcategory that emerged from the PCD data. Students’ learning gains are explained and quotations from the interviews are added to clarify each subcategory of self-reported learning gains.

• (1) Disciplinary conceptual and procedural knowledge gains

This category could be divided into two elements: Theoretical knowledge, and Applying theory in models/graphs/programmes, which are explained hereafter.

Theoretical knowledge

Six students reported on theoretical knowledge in their PCD, naming it, for example ‘theoretical knowledge’ and ‘pure theory. In the interviews, students explained that theory was interesting and that they had learned about different elements of their studies. Mike (Mechanical Engineering), for example, mentioned theoretical knowledge which he had learned, such as understanding of statics, dynamics, liquid dynamics etc. In addition, Ann (Sustainable Innovation) mentioned that she now uses concepts to explain sustainability to others, which she had learned in her studies.

Ann (Sustainable Innovation):

‘For example, sustainability. People are often not sure what that really is. And then I explain it. And use the three things I have learned. That economics, ecology and social, that these three aspects are together sustainable’.

Applying theory in models/graphs/programmes

In their PCD, eight students mentioned having learned how to apply theory, for example, by using mathematical models, developing graphs, and programming. In their interviews, the students explained that the application of theory enabled them to integrate their knowledge and to remember the theory they learned. Steven (Mechanical Engineering), for example, addressed models in which he needed to calculate how much a solar panel would produce and how he could make it as efficient as possible. Peter (Applied Mathematics) mentioned that he had learned about the meaning of certain mathematical problems by developing models with a link to reality. He provided an example of a course in which the students needed to make graphs about prey and predators. His group had chosen a buzzard, wood pigeon, and beetle, and needed to calculate whether these animals could live together in nature, or that one became extinct. They made different models and were able to develop a model in which these animals could survive. Peter explained:

Peter (Applied Mathematics):

‘A lot of times, you have numbers and you do not know where they come from. (…) And now you could see more: “What is the idea behind these numbers and how realistic is this”’.

Peter also mentioned he participated in a course in which he had learned to work with several programmes. The students had to select a part of a city of which data were available in a database and needed to combine the data to make a model of the city via a programme.

Peter (Applied Mathematics):

‘You not only learned to work with one programme, but with several. In addition, you learned to apply the ideas behind the programme, which could be applied to other programmes as well’.

• (2) General cognitive learning gains

Within this category, we could identify two elements: Critical thinking and Inquiry/Design, which are clarified subsequently.

Critical thinking

Four students described learning gains regarding the development of a critical attitude in their PCD. They explained, for example, to have learned not to just presume something but to provide proof for it. Kim (Applied Mathematics) explained that in high school she did calculations, but at the university she was stimulated to think more about the meaning of the calculations and whether they were allowed. Flora (Sustainable Innovation) mentioned that she could not do complicated calculations, because she had not learned that in her studies, but she could understand them and ask critical questions about them. Charlotte (Applied Mathematics) explained:

Charlotte (Applied Mathematics):

‘For example, an even number plus an even number always leads to an even number. But does that always apply? Or is there maybe a number, far away from here, for which this does not apply? It is about: observing critically. And being critical about evidence. And how you are going to prove something’.

Inquiry/Design

Four students mentioned learning gains in their PCD regarding inquiry and design, such as analytical investigations, ideation, and Scrum (product development and testing). In his interview, Walter (Computer Science and Engineering) explained to have learned about Scrum, which refers to a specific process of product development for a client. In addition, Irene (Sustainable Innovation) explained taking apart a product during a course and investigating the origin of each material. The students had to count how much energy was used to make the product. Irene explained to have learned:

Irene (Sustainable Innovation):

‘How you can do good research. So where does a material come from and where can you find it. For example, via internet or by calling a few companies and asking: Where do your raw materials come from?’

• (3) Gains in affect, thought and learning

In this category, we could identify three elements: Self-direction and responsibilities, Ethics, and Taking into account the social context, which are explained hereafter.

Self-direction and responsibilities

Four students referred to learning gains in their PCD regarding responsibility for planning and deadlines, and self-direction. In his interview, Mike (Mechanical Engineering) explained having learned to taking responsibility for a task and making sure that it is finished on time. Steven (Mechanical Engineering) provided a similar explanation.

Steven (Mechanical Engineering):

‘You have responsibility. You can’t hand in nothing or hand it in too late, because people depend on it. And you can’t mess it up, because you will work along further with it’.

In addition, Emma (Industrial Design) addressed in her interview that she had learned from projects in which the students could direct their own learning.

Ethics

The learning gains in this category were mentioned by three interviewed students in their PCD. In the interview, Charlotte (Applied Mathematics) explained a dilemma that had been discussed in a course regarding a self-driving car and had increased her understanding of ethics.

Charlotte (Applied Mathematics):

‘When you collide with your car into a bus, the result could be that there are people with mild injuries. Another option is to evade the collision. When you succeed, there would be no injuries. But when you do hit the bus after trying to evade, there would be people with severe injuries. The question of the producer is: How will you programme the car?’

Charlotte explained that students chose different options to programme the car in the example. This made her become aware of the importance of thinking about ethics. Ann (Sustainable Innovation) mentioned an example about a self-driving car as well. In addition, she explained that in the first or second world war, people worked on hydrogen technology, but this technology could also be used to make explosives for the war. These examples were provided at the technical university and developed her awareness of ethical dilemmas regarding technical innovations.

Taking into account the social context

In their PCD, three students considered the user, differences in cultures, sustainable development and working socially responsible. For example, Rachel (Industrial Design) explained that there were many courses at her faculty that included the user. Students in these courses were asked to have interviews with customers in their field of study. She feels that this is valuable, because students realise how large the difference between their design and the wishes of the user can be. When asked to further explain what she had learned, Rachel provided an example about a head phone she had designed during a project.

Rachel (Industrial Design):

‘We designed a kind of head phone. And it was nicely made, but very awkward in its use. The details were not worked out well. While we had thought it was good. And when you let someone use it, who has never seen this thing before, and does not know how it should be used, that is a realisation of: “Right, this is actually not convenient”’.

When asked about her learning gains, Ann (Sustainable Innovation) mentioned that she would take into account how people interact in other countries. She had worked with other students on a project for a partner in India and explained that the students needed to consider social aspects and how people in India interact with each other. Similarly, Emma (Industrial Design) explained to have learned how to consider different cultures in her designs.

Emma (Industrial Design):

‘How different cultures interpret design and what you have to take into account to make something for different cultures. When you make food for the Netherlands and you take it into India, that will probably not work. Because they have another way of living’.

• (4) Learning gains in teamwork and communication

In this category, two elements could be identified: Teamwork and Communication, which are described hereafter.

Teamwork

Eight students mentioned group work, group dynamics and group roles in their PCD. They explained in their interviews that they worked together in projects and when doing their homework, and that they had learned how to act professionally during group work. Mandy (Industrial Design), for example, explained to have learned how to work together with other students and how to deal with people who might not finish things on time:

Mandy (Industrial Design):

‘I think you learn it [teamwork] mostly by experience. (…) When I was together with my project group for the first time, we decided to talk about ourselves first. To see what kind of people we are. There was one person, for example, who said: “I am chaotic. I am mostly late”. So we can say something about it now, for example: “Pay attention to that”. That worked really well’.

In addition, Flora (Sustainable Innovation) addressed that she had learned to provide feedback to other group members. Irene (Sustainable Innovation) had noticed that the team members of her group first divided the work into different tasks and individually took on a specific task. Later they started to help each other. Irene mentioned having learned that this support contributed to the progress of the project.

Communication

Six students mentioned learning gains that included presenting and writing a report in their PCD. Some students described these skills as ‘soft skills’. Steven (Mechanical Engineering), for example, mentioned that he preferred to present in Dutch, but had learned to present in English as well.

Steven (Mechanical Engineering):

‘In the beginning, I was not very good at it and did not like it at all. And now, it is like, a presentation in English: ok, here we go. (…). I will be able to get the message across’.

In addition, Walter (Computer Science and Engineering) explained that he had learned academic English writing. He mentioned a project in which each team member had to write about his or her specific task within the project. In addition, he addressed homework assignments which included writing skills.

Walter (Computer Science and Engineering):

‘In one of the design-based learning projects we had to do a report. Also, there were some courses that we had to do where we had to write homework. You needed to be specific, and write in an academic kind of way. So, I had to practice on that’.

• (5) Entrepreneurial learning gains

In this category, the main element was Experience and collaboration with companies, which is explained subsequently.

Experience and collaboration with companies

Four students referred to experience and communication with companies in their PCD. In the interviews, students explained it helped them to understand the way that businesses work. According to Kim (Applied Mathematics), the communication with companies was not something she wanted to do in the past. However, she was in a smaller group in a particular course and everyone needed to do the same amount of work. This stimulated her to contact companies as well. James (Sustainable Innovation) explained that he had learned that a company is mostly interested in what they can do with the information that he provided instead of the background of the information. The collaboration with and the case for the company had contributed the most to connecting to the business world according to James.

James (Sustainable Innovation):

‘You can do a lot of research, but in a company they want to know, not what is behind it, but what can I do with it’.

Other results

Seven students mentioned in their interviews that they had learned from practical experience and understanding how their knowledge and skills could be used in a future work field. For example, James (Sustainable Innovation) explained that he had learned frameworks, but these did not have a direct application. He feels that in the business world, the frameworks will be less useful than strategies that can be applied. What works best for him is first receiving theory and then applying the theory in a case. Peter (Applied Mathematics) also described that it was important for him to see how he could apply theory, not only in his studies, but also in future professions.

Peter (Applied Mathematics):

‘Many mathematical studies are often quite theoretical and focused on calculations. It is not always very clear to me to see: later I have an employer, and how do you do this then? (…) So, I like it when it is practice-oriented: “Ok, this you can do with it in the future”’.

5. Discussion and conclusions

In the current study, we addressed the following research question: Which learning gains do students report to perceive in higher engineering education that is changing from teacher-centered to student-centered approaches?

To answer this question, we interviewed thirteen students and asked them about their self-perceived learning gains in higher engineering education. In addition, we asked them to provide an overview of their learning gains in a PCD. The students reported on having learned: (1) theoretical knowledge and application of theory in models, graphs or programmes, which we categorised as disciplinary conceptual and procedural knowledge; (2) critical thinking and problem-solving via inquiry or design, which we categorised as general cognitive learning; (3) self-direction and responsibility, being aware of ethics, and taking into account the social context, which we categorised as affect, thought and learning; (4) working in a team and communicating with other students, which we categorised as teamwork and communication; and (5) experience and collaboration with companies, which we categorised as entrepreneurial learning. We define these identified categories as ‘strands’, as they are interwoven. Each strand consists of learning gains the participants of the current study perceived during their time at the technical university. In the following paragraphs, each strand is explained, and connected to and elaborated with relevant literature.

The first strand of learning gains in higher engineering education which we distinguished is the disciplinary conceptual and procedural knowledge strand. This strand refers to developing theoretical knowledge. For example, knowledge about concepts in science, mathematics and fundamental engineering, such as statics and dynamics. Moreover, this strand includes gaining subject matter knowledge of relevant (engineering) topics, such as knowledge about sustainability. Students explained that they had learned theoretical knowledge in different courses during their studies and were able to talk about the concepts they had learned. Improved skills in applying theoretical knowledge in models, graphs and programmes is also part of this strand. Students had learned to use different programmes in which they could apply their knowledge, and made models and graphs to interpret data and solve problems. This strand of learning gains match learning gains described by Rogaten et al. (2019), such as discipline-specific skills, and knowledge and understanding of the topic. In addition, it matches knowledge and skills mentioned by Purzer, Fila, and Nataraja (2016) regarding math, science and technical knowledge, and technical skills; and by Crawley et al. (2011), such as knowledge of underlying mathematics and sciences, and core engineering fundamental knowledge.

As a second strand, we identified the general cognitive learning strand. This strand includes acquiring (twenty-first century) skills, such as critical thinking. Students explained that they had learned to ask critical questions and that they were more aware of the need to provide evidence, instead of presuming something. Other cognitive learning skills addressed in the literature to take into account are, for example, analytical reasoning, problem-solving, system thinking, and creative thinking (Crawley et al. 2011; Purzer, Fila, and Nataraja 2016; Rogaten et al. 2019). In addition, this strand refers to learning gains regarding inquiry and design processes, such as conducting investigations and learning about Scrum. This matches learning gains and outcomes mentioned in the literature, such as research skills and knowledge (Rogaten et al. 2019); ability to create unique solutions to current problems (Purzer, Fila, and Nataraja 2016); experimentation, investigation and knowledge discovery; and conceiving (forming an idea and setting goals), designing, implementing, (training and managing of) operating (Crawley et al. 2011).

A third strand that we distinguished is the affect, thought and learning strand. This strand concerns improving students’ ability to direct their own learning and being responsible for planning and deadlines. For example, making sure that a task is finished on time. In literature, additional skills and attitudes are described, such as enthusiasm for or interest in a topic, motivation, self-efficacy and persistence (Purzer, Fila, and Nataraja 2016; Rogaten et al. 2019). Moreover, this strand involves the development of ethics. For example, by thinking about ethical dilemmas regarding the field of engineering. This is recognised as an important learning gain or outcome in literature as well (Crawley et al. 2011). Finally, this strand includes learning to take into account the social context, such as the user of technical products and social differences between cultures. For example, by not only thinking about engineering elements, but also to consider the wishes of users. This is comparable to literature that includes learning to orientate towards customer needs (Purzer, Fila, and Nataraja 2016) and taking into account the external, societal and environmental context (Crawley et al. 2011).

As a fourth strand, we defined the teamwork and communication strand. This strand refers to learning how to work in a team, for example, how to act professionally during group work and provide feedback to team members. In literature, learning gains regarding team work skills are mentioned as well (Crawley et al. 2011; Purzer, Fila, and Nataraja 2016; Rogaten et al. 2019). In addition, this strand includes developing communication skills, such as presenting and writing a report. For example, learning to present in English, when English is not the students’ first language, or learning to write in an academic way. This connects to developing communication skills in literature (Crawley et al. 2011; Purzer, Fila, and Nataraja 2016; Rogaten et al. 2019).

Finally, as a fifth strand, we distinguished the entrepreneurial learning strand which refers to experience and collaboration with companies. The students had learned to contact companies and about the way that businesses work. This matches literature regarding learning about the enterprise and business context (Crawley et al. 2011), and organising and managing projects (and businesses) (Purzer, Fila, and Nataraja 2016).

To further clarify how the strands in our study compare to types of learning objectives, outcomes and gains in literature, we provide an overview of the taxonomy of educational objectives of Bloom (1956), the four pillars of education of Delors (1996), the learning gains framework of Vermunt, Ilie, and Vignoles (2018), and important competences in engineering education (Leandro Cruz, Saunders-Smits, and Groen 2020) in relation to the strands of our study (see Table 2).

Table 2. Relation of the five strands to types of learning objectives, outcomes and gains.

	Strands of learning gains	Taxonomy of educational objectives (Bloom 1956)	Four pillars of education (Delors 1996)	Learning gains framework higher education (Vermunt, Ilie, and Vignoles 2018)	Competences in engineering education (Leandro Cruz, Saunders-Smits, and Groen 2020)
1	Disciplinary conceptual and procedural knowledge	Cognitive domain	Learning to know	–	–
2	General cognitive learning	Cognitive domain	Learning to know	Cognitive component	Innovation/creativity
3	Affect, thought and learning	Affective domain	Learning to be Learning to live together	Metacognitive component Affective component	Lifelong learning
4	Teamwork and communication	Affective domain Behavioural domain	Learning to live together Learning to do	The socio-communicative component	Communication Teamwork
5	Entrepreneurial learning	Affective domain Behavioural domain	Learning to live together Learning to do	–	–

Table 2 shows that the types of learning objectives, outcomes and gains match the strands that we identified in our study. The strands specifically refer to the field of engineering education and address students’ perceived learning gains. However, the first and fifth strand do not appear in all of the studies in Table 2. Our research shows that, in addition to general cognitive learning; affect, thought and learning; and teamwork and communication, students perceived both disciplinary conceptual/procedural and entrepreneurial learning gains during their higher engineering education.

In addition to the knowledge and skills regarding the different strands, students explained having learned from practical experience and understanding how knowledge and skills can be applied in a future profession. This implies students’ preference for student-centered approaches in which they can apply their knowledge and gain insight into how their knowledge and skills can be used after their graduation as an engineer.

The current study has provided insights into students’ perceived learning gains in higher engineering education that is changing from teacher-centered to student-centered approaches. The qualitative data regarding the students’ PCDs and interviews revealed specific strands of learning gains. We claim that this can be viewed as a first step towards framing learning gains in higher engineering education. Quantitative studies would have to be conducted in order to test these results for their reliability and generalisability, and hence the framing of student learning gains we propose.

5.1. Limitations and recommendations for future research

In the current study, we investigated students’ learning gains via interviews and PCDs. A possible limitation is that the students’ individual course syllabi focused on areas that the students described in their PCD. However, the students were participating in different educational programmes and had the possibility to choose different courses and modules within these programmes. Furthermore, we chose participants for our study who participated in student-centered courses in addition to teacher-centered courses. Therefore, the learning gains in their PCDs are based on a varied educational background in higher engineering education.

We determined students’ learning gains via self-reports in the current study, which could be considered a limitation. Although self-reports reveal students’ individual perspective on their learning gains (Baume 2018), a combination of self-reports and other measures, such as rubrics or pre-tests and post-tests would have provided a more thorough insight into students’ learning gains.

Another possible limitation of the current study is that we included only 13 participants. However, this is suitable for our qualitative study in which we developed an in-depth view on students’ self-reported learning gains. For future research, we recommend to investigate larger groups of students. That would imply developing quantitative measurement instruments. Inspiration for these measurement instruments can be found in the PCDs that we used in the current study, and can come from research on general learning gains in higher education (see Vermunt, Ilie, and Vignoles 2018) and measurement of specific learning gains (e.g. Aloisi and Callaghan 2018; Arico et al. 2018; Turner et al. 2018).

The learning gains of participants of our study were acquired in the context of the technical university at which the research was conducted, which could be considered a limitation of the current study. A recommendation for future research would be to investigate students’ learning gains at other technical universities (and different cultural spheres) to get insight into students’ perceived learning gains in different contexts.

5.2. Recommendations for practice and policy

The current study revealed strands of learning gains that emerged from the students’ interview data and PCDs. Educators could take these strands of learning gains and elements within each strand into account when developing a curriculum or a specific course within the curriculum. For example, the students in the current study explained their perceived entrepreneurial learning gains, and what they had learned about the way that businesses work by actually being in contact with these companies. In addition, the students mentioned to perceive conceptual and procedural knowledge gains when learning how to apply their disciplinary knowledge into graphs, models and programmes. The practical experience was appreciated by the students, because it helped them to understand how to use their knowledge in a future profession. These aspects can inform educators about the importance of including practical experience in their education instead of just providing lecture-based education.

In addition to including the five strands in their courses, educators are recommended to assess learning gains regarding these different strands, for example via rubrics. Furthermore, it is important for educators to make students’ own perspectives on their learning gains explicit as these might influence students’ choices for subjects or courses in higher education. For example, by asking students to describe what they have learned at the end of a course and what they would like to further develop. Or by inviting them to make a PCD to get an overview of their perceived gains. This can provide students with insight into their own learning and areas they are interested in and/or would like to further improve.

The current study shows which elements the participants highlighted as their learning gains during their time at the university. However, the students might have developed certain competences which they are not aware of. In the current study, for example, creative thinking was mentioned in the literature related to the general cognitive learning strand, but this cognitive learning skill was not addressed as a learning gain by the students. When certain competences are important during students’ education or for their future profession, educators might address these more specifically in their courses. In that way, students can make more informed decisions during their higher engineering education and can highlight specific competences when applying for jobs in the field of engineering.

The identified strands provide a first view on students’ perceived learning gains in higher engineering education that includes student-centered approaches, such as design-, and challenge-based learning. A recommendation for higher education in general is to take students’ own perception of what they learn into account when developing or enhancing the curriculum.

Ethical approval

Ethical approval was granted by the Ethical Review Board of Eindhoven University of Technology in the Netherlands.

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

This study has been financially supported by the Innovation Fund of 4TU.CEE at Eindhoven University of Technology.

Notes on contributors

Martina S. J. van Uum

Martina van Uum is an assistant professor at Radboud Teachers Academy of Radboud University in the Netherlands. Her research focuses on innovative educational pedagogies, such as inquiry-based and challenge-based learning, and education that includes creativity.

Birgit Pepin

Birgit Pepin is a full Professor of Mathematics/STEM Education at Eindhoven School of Education of Eindhoven University of Technology in the Netherlands. Her teaching and research focus on (digital) STEM curriculum materials and their design/use by teachers or students, and teacher professional learning in STEM education (in secondary and tertiary/higher education).