From CDIO to challenge-based learning experiences – expanding student learning as well as societal impact?

ABSTRACT

Challenge-based learning (CBL) is a multidisciplinary approach that encourages students to work actively with peers, teachers and stakeholders in society to identify complex challenges, formulate relevant questions and take action for sustainable development. In this paper, it is argued that CBL can be viewed as an evolution of the Conceive, Design, Implement, Operate concept, expanding as well as deepening the learning experience. The study reported on investigates the multiple aims of a particular CBL environment (the Challenge Lab at Chalmers University of Technology), which are to combine significant student learning and societal transformation. The results show that the students perceive that they have developed deep skills in problem formulation and sustainable development, as well as working across disciplines and with different stakeholders. Moreover, the study shows that although few student projects reach implementation stage, there is a potential for societal impact both during and after the Challenge Lab learning experience.

KEYWORDS:

• Challenge-based learning
• CDIO
• sustainability
• multidisciplinary learning
• stakeholder involvement

Introduction

Recently, the idea of ‘grand challenges’, i.e. issues that critically need to be addressed by society to improve for humankind during the coming century, has been very influential in defining directions and scope for national research and innovation agendas (European Commission 2014; NAE 2008). The concept has also inspired many novel educational developments to describe themselves as ‘challenge-based’ learning experiences (Malmqvist, Kohn Rådberg, and Lundqvist 2015). Malmqvist et al. propose the following definition:

Challenge-based learning takes places through the identification, analysis and design of a solution to a sociotechnical problem. The learning experience is typically multidisciplinary, involves different stakeholder perspectives, and aims to find a collaboratively developed solution, which is environmentally, socially and economically sustainable.

There are many potential benefits associated with challenge-based learning (CBL) experiences. They address several of the key features of future, globally leading engineering programmes as identified by Graham (2017), e.g. embracing authentic, active learning; offering choice in problem solving and learning practices and enabling training in multidisciplinary teamwork and decision-making; as well as harnessing the desire of many students for a sense of meaning in their education. CBL experiences can be seen as a further step in enhancing students’ use of design as ‘learning through’ rather than ‘learning to’ (Bernard, Edström, and Kolmos 2016) as well as a step in integrating sustainability in engineering education (Enelund et al. 2013). In a previous study, we surveyed and compared a number of examples from across the world, showing that the concept can be implemented across several study years, that it can be part of the curriculum as well as be extracurricular, and address a fairly small as well as very large student cohort (Malmqvist, Kohn Rådberg, and Lundqvist 2015).

A main precursor to CBL experiences is problem-based learning, where teams of students are posed with a design, research or diagnostic ‘problem’, and the learning takes place through the process of working out the solution (Hmelo-Silver 2004; Malmqvist, Kohn Rådberg, and Lundqvist 2015). Another main precursor to CBL is the Conceive, Design, Implement, Operate (CDIO) approach to engineering education (Crawley et al. 2014). The CDIO approach emphasises the importance of learning experiences where products that meet customer needs are developed through a process of conception, design, implementation and operation. However, in contrast to CDIO, CBL experiences can be argued to expand the scope and depth and bring forward certain additional characteristics – for example, by including tasks of problem identification and formulation, as well as dialogues with core stakeholders, and by adding business model components to engineering solutions, and considering the societal context and impact of a product rather than just the corporate benefits. In addition, CBL experiences also seek to foster the ability of teamwork, and personal awareness, by considering ‘values’ and ethics in addition to customer needs in decision-making. The development of traditional and CDIO/problem-based learning to CBL is outlined in Figure 1.

Figure 1. Evolution from traditional to problem-based to challenge-based education (Malmqvist, Kohn Rådberg, and Lundqvist 2015).

In addition to student learning, CBL experiences and the environments in which they take place may offer the possibility of serving a broader purpose and contribute to added values for the society. In describing the CBL environment that will be investigated in this study, the Chalmers Challenge Lab, Larsson and Holmberg (Forthcoming) highlight the importance of transformation and integration in accomplishing sustainable development and investigate how challenge-based learning environments can create transformative and integrative value inside as well as outside higher education institutions. They describe transformative value as ‘outcomes that challenge business-as-usual practices that reinforce the lock-in of systems understood as unsustainable’, and integrative value as ‘awareness raised and trust built when a diverse group of actors, disciplines, and perspectives are brought together in dialogue to explore a common issue’. Furthermore, Larsson and Holmberg (Forthcoming) distinguish between value creation in terms of direct outcome and value creation in terms of potential future societal impact.

From an educational perspective, it is important that such added values and multiple aims of CBL experiences do not occur at the expense of student learning, which could result in students not meeting the learning requirements for their education. Some potential obstacles for students to meet the required learning outcomes could be that the research questions investigated by students engaging in CBL experiences might not reach the required technical depth and that much time is spent on interactions with actors in society.

Accordingly, the aim of this study is to explore some aspects of student learning in the context of a CBL environment, using the Chalmers Challenge Lab as a case study. The objectives are:

• to assess whether the students’ self-perceived learning fulfils the required learning outcomes for MSc theses at Chalmers in general, i.e. the required academic learning outcomes for their education;

• to identify some potential additional learning outcomes that the students perceive that they have achieved, which are not developed to the same extent in traditional MSc theses;

• to assess how much the MSc thesis projects at Challenge Lab have achieved in the pathway from problem to implementation, in line with the CDIO cycle, and to discuss this in relation to the ambition to create transformative and integrative values for society.

The remainder of the paper is structured as follows: first, the study is positioned in relation to (1) earlier studies and movements in the area of social labs and innovation labs; and (2) format and learning outcomes of engineering and sustainability education, both areas closely connected to the notion of CBL. This is followed by a description of the research methods used and the empirical setting for the study (Chalmers Challenge Lab), followed by results, and finally a discussion of findings.

Conceptual framework

Social and innovation labs

CBL experiences, placed in a societal context, have benefited from and driven the emergence of a new kind of learning environment, the ‘social’, or ‘open innovation’, lab. In a survey paper, Gryszkiewicz, Lykourentzou, and Toivonen (2016) define an (open) innovation lab as a ‘semi-autonomous organisation that engages diverse participants – on a long-term basis – in open collaboration for the purpose of creating, elaborating, and prototyping radical solutions to open-ended systemic challenges’. ‘Social labs’ are further characterised by Hassan (2014) as social (the actors actively participate, not just as experts but as co-creators), experimental (solutions are developed and prototyped in an iterative process) and systemic (solutions should not only mitigate symptoms or parts of problems but aim to identify and address the root cause of the problems). Examples of such labs are the Vancouver CityStudio (Moore, Van Wynsberghe, and Holden 2007), which focuses on urban development, the Royal Institute of Technology (KTH), Stockholm, OpenLab (Berglund and Bernhard 2015), which leans more strongly towards multidisciplinary business and product development in Information Technology (IT) and the Life Sciences, and Chalmers Challenge Lab (Holmberg 2014; Kohn Rådberg 2014), which focuses on specific aspects of urban development and is described in greater detail below.

These learning environments have in common that they are designed as arenas where university, business and the public sector collaborate around common issues. They aim not only to foster student learning, but also to promote collaboration between partners and to create societal and technical solutions to difficult, strategic challenges. They are often located outside or on the periphery of university campuses in order to serve as a ‘neutral ground’ between stakeholders. Such ‘neutral ground’ enables a more open dialogue and fosters mutual understanding of each other’s perspectives and work, creating a common ground for collaboration between the stakeholders who otherwise tend to compete over what view or logic to base the collaboration on (Edelenbos, Bressers, and Schloten 2016). Students play a key role in these environments, not only in problem-solving, but also in driving a collaborative, multi-perspective dialogue on defining the problem to be solved. In addition to a physical arena, the labs typically provide a set of methods for addressing a societal challenge, from problem identification to solution concept.

Format and learning outcomes

The case investigated in this study of CBL (i.e. the Chalmers Challenge Lab) is also conceptually situated in the two contexts of engineering education and sustainability education – both inputting to the format and learning outcomes of CBL.

In general, the required learning outcomes of engineering education naturally evolve in relation to the professional role of the contemporary engineer, and recent calls for redirection of engineering education emphasise the need for engineering graduates with more solid skills in communication, collaboration, life-long learning, leadership, creativity, entrepreneurship, sustainability and ethics (Crawley et al. 2014; Huntzinger et al. 2007; Splitt 2003).

At a detailed level, the CDIO syllabus 2.0 offers a comprehensive reference list of the competences expected from graduated engineers (Crawley et al. 2014). The CDIO syllabus 2.0 (Table 1) proposes that engineering competences can be divided into four main categories: (1) Disciplinary knowledge and reasoning; (2) Personal and professional skills and attributes; (3) Interpersonal skills: teamwork and communication; and (4) Conceiving, designing, implementing and operating systems in the enterprise, societal and environmental context. The categories are further detailed into more specific topics, as shown in Table 1.

Table 1. Conceive, design, implement, operate (CDIO) syllabus 2.0 at the second level of detail (Crawley et al. 2014).

1 DISCIPLINARY KNOWLEDGE AND REASONING 1.1 KNOWLEDGE OF UNDERLYING MATHEMATICS AND SCIENCE 1.2 CORE FUNDAMENTAL KNOWLEDGE OF ENGINEERING 1.3 ADVANCED ENGINEERING FUNDAMENTAL KNOWLEDGE, METHODS AND TOOLS 2 PERSONAL AND PROFESSIONAL SKILLS AND ATTRIBUTES 2.1 ANALYTICAL REASONING AND PROBLEM SOLVING 2.2 EXPERIMENTATION, INVESTIGATION AND KNOWLEDGE DISCOVERY 2.3 SYSTEM THINKING 2.4 ATTITUDES, THOUGHT AND LEARNING 2.5 ETHICS, EQUITY AND OTHER RESPONSIBILITIES

3 INTERPERSONAL SKILLS: TEAMWORK AND COMMUNICATION 3.1 TEAMWORK 3.2 COMMUNICATIONS 3.3 COMMUNICATION IN FOREIGN LANGUAGES 4 CONCEIVING, DESIGNING, IMPLEMENTING, AND OPERATING SYSTEMS IN THE ENTERPRISE, SOCIETAL AND ENVIRONMENTAL CONTEXT 4.1 EXTERNAL, SOCIETAL AND ENVIRONMENTAL CONTEXT 4.2 ENTERPRISE AND BUSINESS CONTEXT 4.3 CONCEIVING, SYSTEMS ENGINEERING AND MANAGEMENT 4.4 DESIGNING 4.5 IMPLEMENTING 4.6 OPERATING

The full CDIO syllabus 2.0 comprises four to five levels of detail. It has been shown (Crawley et al. 2014) to encompass all competencies listed in various international and national accreditation requirements (ABET (2015); EUR-ACE (ENAEE 2015), etc). One example of such a standard is the Swedish Higher Education Degree Ordinance, which identifies 12 learning outcomes that an engineering graduate should be able to demonstrate, including ‘ability to create, analyse and critically evaluate different technical solutions’ (Ministry of Education 2006). The national intended learning outcomes (ILOs) in Sweden are used later in this paper to assess the student learning that took place during the studied CBL experience.

In the context of sustainability education, a large number of descriptive and prescriptive studies regarding both approach and intended outcomes of sustainability education have been conducted during the last decades (Azapagic, Perdan, and Shallcross 2005; De Haan 2006; Enelund et al. 2013; Mogensen and Schnack 2010; Segalàs et al. 2009; Sipos, Battisti, and Grimm 2008; Svanström, Lozano-García, and Rowe 2008), with much focus given specifically to engineering education in 2005–2009 (Wu and Shen 2016). This has resulted in a wide range of pedagogical practices being implemented for sustainability in engineering education, focusing on ‘student-centred and interactive enquiry-based approaches’ (Trimingham et al. 2016, 179). In the CDIO syllabus 2.0, sustainability-related learning topics include Sustainability and the Need for Sustainable Development: Definition, Principles, Goals and Importance of Sustainability (item 4.1.7) and Design for Sustainability (item 4.4.6) and several additional topics. According to the Swedish Higher Education Degree Ordinance, graduating engineers are expected to demonstrate ‘insight into the possibilities and limitations of technology, its role in society and people’s responsibility for how it is used, including social and economic aspects as well as environmental and safety aspects’ (Ministry of Education 2006).

In the later stages of research into the format of sustainability education, an emergent advocacy towards project- and problem-based approaches (e.g. Brundiers and Wiek 2013; Lehmann et al. 2008; Wiek et al. 2014), experiential approaches (Caniglia et al. 2016; Savage et al. 2015) and competence-based education (e.g. McAloone 2007; Mochizuki and Fadeeva 2010) can be seen. Under the lens of constructive alignment (Biggs 1996), this coevolution is natural – as an ambition to develop integrative competences (encompassing knowledge, skills, attitudes, etc.), rather than stand-alone demonstrable knowledge and skills. In the specific case of achieving development of sustainability competences, which is relevant across different educational formats for sustainability education, it has moreover been indicated that learning experiences should focus on the interplay between cognitive and non-cognitive (affective, conative) components, have a social orientation and be student-centred (Barth et al. 2007).

Synthesising much of the debate on learning outcomes in sustainability education, Wiek, Withycombe, and Redman (2011) developed a framework for key competences to be used in sustainability problem-solving, including systems thinking, and anticipatory, normative, strategic and interpersonal competence. Further, in a Delphi study consensus on systemic thinking and handling of complexity, Rieckmann (2012) established anticipatory thinking and critical thinking as the most important competences for sustainability.

Method

The research presented here is based on a longitudinal case study (Eisenhardt 1989; Yin 2013) of Chalmers Challenge Lab. Data were collected over 3 years, 2014–2016, and represent three different student groups.

To gain insight into student learning, a qualitative approach was chosen, where the data were primarily collected through individual semi-structured, in-depth interviews (Fontana and Frey 2000; Kvale 2008). Some of the researchers were active as teachers, project leaders and thesis supervisors in the Challenge Lab, while at the same time conducting the study. Such close connection to the case studied offers the advantage of having insight into and access to the organisation. However, researchers who did not have a connection to the Challenge Lab were also involved in the study. Furthermore, in order to safeguard against biased, taken-for-granted patterns and the risk of ‘going/staying native’ (Alvesson 1999), discussions were held with senior researchers at the faculty. We further constantly challenged the interpretations of the respondents’ accounts by inviting fellow researchers to assist in the interpretations. One advantage of being an insider lies in contextual understanding, contributing significantly to the quality of the knowledge generated (Bartunek and Louis 1996).

The interviews consisted of 12 predefined questions, leaving room for follow-up questions. The questions were related to project preparation, project execution, project and learning evaluation and the effects of the above on the student’s future career. During the interviews, in addition to the questions, the students were asked to conduct a self-assessment of how the thesis project had advanced their mastery of the ILOs of Chalmers MSc thesis projects. The assessment was made in connection with the interviews, but was not a part of them, as the students completed the questionnaire individually. The answers to 12 questions were given on a graded scale from 1 to 5, where 1 was ‘not at all’ and 5 was ‘very significantly’. Altogether 37 out of 38 students completed the questionnaire. The questionnaire largely reflected the full set of ILOs of MSc (Engineering) degrees in Sweden (Ministry of Education 2006). The interviews were carried out by two interviewers and lasted for about 1.5 hours. Out of 38 students, 37 were interviewed: 11 in 2014, 13 in 2015 and 13 in 2016. Furthermore, the Challenge Lab students’ self-assessments (see above) were analysed and compared with the results from a questionnaire sent in 2016 to all 1765 students at Chalmers who performed their MSc thesis in that academic year. The response rate was 27% (i.e. 479 students). The purpose, here, of including the analysis of the self-assessment, in addition to the qualitative data, is to further support the analysis and illustrative quotations.

In the context of direct transformative value, it was investigated how far the MSc thesis projects have come towards reaching implementation phase and/or bringing real change. To this end, the results were classified with respect to their coverage of a typical problem to implementation pathway, similar to the CDIO cycle though excluding the operation phase. The pathway starts with: (1) problem formulation, and is followed by the phases: (2) idea or model generation; (3) concept development; and (4) test/evaluation within an academic setting. It concludes with (5) testing/evaluation by external stakeholders.

Empirical setting – Chalmers Challenge Lab

The goal of Chalmers Challenge Lab is to establish an arena where students interact with actors in society, with the aim to create student learning as well as transformative and integrative values for sustainable development. The Challenge Lab aims to address the complex societal sustainability challenges, through the integration of multiple perspectives of relevant stakeholders as well as sustainability issues to enhance a societal transformation towards sustainability (Holmberg 2014; Kohn Rådberg 2014; Kohn Rådberg et al. 2015; Larsson and Holmberg Forthcoming). The Challenge Lab is based on a back-casting approach starting from sustainability principles (Holmberg 1998). It is further based on the recognition that students can play an important role in supporting ‘transitions’ for sustainability (Geels 2002; Loorbach 2007). Therefore, relevant methods and perspectives are taught and offered to the students in the Lab, supporting them in developing capacities to take on transformative and integrative research questions in their projects.

Challenge Lab can be compared to ‘social labs’ (Hassan 2014), being social, i.e. with the actors actively participating, not just as experts but as co-creators, experimental, i.e. with solutions that are developed and prototyped in an iterative process, and systemic, i.e. with the aim that solutions not only mitigate symptoms or parts of the problems but identify and address the root cause of the problems. However, some additional specific features of Chalmers’ Challenge Lab which extend the arena beyond that of other social labs are that students spend much effort on formulating sustainability criteria and research questions based on sustainability challenges that they identify when applying a back-casting approach from sustainability principles (Holmberg 2014; Larsson and Holmberg Forthcoming).

The main focus of this study was the MSc thesis course offered at Chalmers’ Challenge Lab, which is open to students from different disciplines, and the CBL experience of students partaking in this course. Chalmers’ Challenge Lab also offers a preparatory course to be undertaken by students before they start their MSc thesis (Figure 2). The preparatory course was developed after the first cohort of students had enrolled in the MSc thesis course, with the aim to provide them with theoretical perspectives, methods and tools used in the MSc thesis course, combined with the opportunity to practise these within a smaller team project. The aim is to prepare students who come from different engineering fields for the course work ahead, which in turn will allow more time to advance the MSc thesis projects later on, towards developing solutions that can be tested in practice, by the students and stakeholders in collaboration. The preparatory course includes knowledge about multi-level complex dynamic systems, sustainability principles and back-casting, as well as design thinking, and it also includes tools for self-leadership and multi-stakeholder interaction (Chalmers University of Technology 2016). The preparatory course is not compulsory for students doing their MSc theses in the Challenge Lab, but it is recommended.

Figure 2. The Challenge Lab process.

The MSc thesis course is structured in two phases. The first phase focuses on problem formulation and the scope of projects, followed by the second phase where students carry out their studies or project. Formulating the problem is recognised as a central part of working on issues and challenges related to sustainability, where perspectives of multiple stakeholders as well as multiple knowledge disciplines need to be taken into account in understanding and framing a problem. In formulating the problem, the students carry out dialogue meetings with the regional public sector, industry and researchers. During these meetings, different perspectives on the problem emerge that guide the work in how to formulate the research question to allow the different perspectives to be included. In the project (second) phase, the students work independently or in pairs on their MSc thesis projects. However, the projects collectively address a yearly theme, and the students work in the same physical space, continually updating and supporting each other’s projects.

The theme for the MSc theses undertaken in the Challenge Lab in 2014 was ‘Urban Transportation’, but since many of the challenges also included other aspects such as energy or the built environment, the theme was extended to ‘Urban Development’ for 2015 and 2016, and related to projects and stakeholders in the Gothenburg Region. The projects therefore leaned towards the regional environment sustainability goals, but also reflected the broad range of disciplines included in the Lab. The titles of the thesis projects are listed in Table 3.

As previously mentioned, the Challenge Lab occupies its own physical space, where the students meet on a daily basis and to which stakeholders are invited for dialogues with the students as well as with other stakeholders. Here the students arrange dialogues and meetings when working on their projects and theses. The common space further allows a deeper collaboration between the students and discussions between different disciplines and projects. The Challenge Lab is located at the intersection of Chalmers and a neighbouring science park in order to create a ‘neutral’ arena where stakeholders from industry, academia and the public sector can meet and work towards addressing long-term challenges. The team of teachers who work in the Lab on a daily basis to support the students reflect the multidisciplinary setup and include senior faculty members from Chalmers’ departments of Space, Earth and Environment, Technology Management and Economics, and Industrial and Materials Science. In addition, supervisors from relevant disciplines are involved in the team process. The supervisors are selected based on the specific subject areas of the theses, to secure the quality of the MSc theses and add deeper knowledge in the specific subject areas of the students.

A pedagogical idea with Challenge Lab is to develop student skills in working across disciplines and from a challenge-driven perspective. This means extending beyond the typical MSc thesis projects, i.e. not only designing a solution, but also identifying critical aspects of a system in order to formulate relevant questions, by integrating the perspective of relevant stakeholders before designing the solution and thereafter also implementing the solution. The specific ILOs of the MSc thesis are to (Chalmers Challenge Lab 2017):

• describe critical sustainability challenges and reflect upon necessary paradigm shifts;

• describe how sustainability challenges affect industrial and societal actors, and how they are interlinked;

• reflect on the challenges of policy implementation for sustainable development;

• apply sustainability criteria for research utilisation and innovation processes;

• reflect upon important ‘lock-ins’ at societal, organisational and individual levels, which are relevant for sustainability challenges;

• apply a systems perspective to meet sustainability challenges;

• apply relevant sustainability frameworks (e.g. back-casting);

• apply practical methods and tools for sustainable product development and design;

• apply basic theories about and tools for transformative leadership in a challenge-driven process;

• apply tools to enable and facilitate dialogue with multiple stakeholders; and

• extract and manage new knowledge in a complex context.

In addition to these specific learning ambitions of Challenge Lab, students are required to achieve the ILOs for all MSc thesis projects at Chalmers, including specialised knowledge within their discipline, the ability to work independently, and written and oral communication skills. In summary, the MSc thesis course at the Challenge Lab at Chalmers offers an advanced, CBL experience, involving students from across the university with ambitions to make a difference locally to create transformative and integrative values towards sustainability. The Challenge Lab has its own dedicated space and is supported by faculty from several departments. It places a specific focus on developing problem identification and formulation skills based on sustainability criteria when applying back-casting from sustainability principles and in dialogues with actors in society. A more elaborated description of the Chalmers Challenge Lab can be found in: Holmberg (2014) and Larsson and Holmberg (Forthcoming) and on their web page (Chalmers Challenge Lab 2017).

Results

In accordance with the objectives of the paper, the presentation of findings is structured in two sections: results related to student learning and results showing how far the MSc theses have come towards reaching implementation.

Student learning

The perceived academic learning of the students who conduct their MSc thesis at the Challenge Lab is in line with the perceived learning of Masters students at Chalmers in general. The self-assessed academic learning of the Challenge Lab students in terms of the ILOs of Chalmers MSc thesis projects is presented in Table 2. The mean values in the self-assessment for all 3 years for all ILOs are at least 3.3 on a 1–5 scale, which is in line with the results from a questionnaire in 2016 sent to all students at Chalmers who performed their MSc thesis in that academic year, in which the lowest mean value was 3.6.

Table 2. Students’ self-assessments of how the project had advanced their mastery of the learning outcomes of Chalmers MSc thesis projects.

Intended learning outcome	2014			2015			2016
	Challenge Lab			Challenge Lab			Challenge Lab			All
	M	SD	R	M	SD	R	M	SD	R	M
1. Specialised knowledge within your main field of study including insight into relevant research and development work	4.2	0.83	2–5	3.7	1.07	2–5	3.9	1.14	2–5	4.2
2. Specialised knowledge of methods within the main field of study of the study programme	3.6	0.92	2–5	3.3	1.45	1–5	4.2	1.03	2–5	4.2
3. Ability to contribute to research and development work	3.3	0.75	2–4	3.9	1.00	2–5	4.3	0.85	3–5	4.1
4. Ability to identify, formulate and manage complex problems in a critical, independent and creative manner from an overall perspective	4.8	0.39	4–5	4.2	0.80	3–5	4.5	0.50	4–5	4.1
5. Ability to plan and use adequate methods to implement advanced tasks within given frameworks, as well as evaluate these efforts	3.9	0.67	3–5	3.6	0.84	2–5	4.2	0.66	3–5	4.2
6. Ability to create, analyse and critically evaluate different technical/architectural solutions	3.4	1.34	1–5	3.4	1.15	1–5	4.2	0.77	3–5	4.0
7. Ability to integrate knowledge in a critical and systematic manner	4.4	0.88	2–5	3.8	1.10	2–5	4.3	0.72	3–5	4.2
8. Ability in writing to clearly describe and discuss your conclusions in English, including the knowledge and arguments that form the basis of the conclusions	3.8	0.57	3–5	3.5	1.15	1–5	4.2	0.77	3–5	4.5
9. Ability to orally clearly describe and discuss your conclusions in English, including the knowledge and arguments that form the basis of the conclusions	4.4	0.66	3–5	3.8	1.05	1–5	4.3	0.82	3–5	4.5
10. Within the framework of the specific degree project, ability to identify which problems need to be addressed to observe sustainable development	4.4	0.77	3–5	4.5	0.63	3–5	4.7	0.46	4–5	3.8
11. Awareness of ethical aspects related to research and development work	3.4	0.83	2–5	3.5	1.22	1–5	3.5	0.84	2–5	3.6
12. Ability to work independently as a holder of a Civilingenjör, Master of Architecture or Master of Science degree	4.5	0.67	3–5	4.0	1.04	1–5	4.4	0.49	4–5	4.1

Notes: The results for the Challenge Lab students are presented in comparison to the results from a questionnaire sent in 2016 to all students at Chalmers who performed their MSc thesis in that academic year. 1 = not at all, 2 = a little, 3 = to some extent, 4 = significantly, 5 = very significantly. M = mean; SD = standard deviation; R = range.

There was a clear difference between the 3 years and also between students in the Challenge Lab with regard to how well they thought they met the ILOs. The last year, 2016, had the largest number of ILOs with a mean value above 4 (10 out of 12 compared with 3 out of 12 for 2015, the year with the lowest number of ILOs rated >4). In 2015, there was a larger difference between the students, as can be seen from the standard deviations which have the higher values in 2015. This can be compared to the results from the questionnaire in 2016 sent out to all students at Chalmers, in which 10 out of the 12 ILOs had a mean value >4.

Specialised knowledge within the main field of study

As students from all disciplines and programmes are invited to apply to Challenge Lab, there has been an emphasis from faculty and supervisors to ensure that each student accomplishes a high level of specialised knowledge in their field of study, even if working cross-disciplinarily. The mean values in the self-assessment of ILOs No. 1 and 2 (both regarding specialised knowledge within the main field of study) (Table 2) can be considered good. However, it should be noted that some students gave low values for these ILOs, and in 2015 the standard deviation for the ILO about specialised knowledge of methods within the main field of study (No. 2) was the highest for all ILOs during the 3 years. There are many examples from the interviews of students talking about different experiences and expressing different views on how the specific knowledge was developed. As one student, who had expected to further develop a specific deep knowledge, expressed:

Early on I understood the challenge to further develop deep knowledge in my area together with others from other areas. Looking at the result I believe it is possible, as I had to develop a deeper understanding of my field in order to develop our common knowledge in the project that was broader. It is just very different. (Student 3)

Others put forward the perspective that specific knowledge is valuable in developing new knowledge across disciplines:

I never thought one could think of a system in so many different ways. I have learned a lot related to my own area just by working together closely with someone who had a different perspective. (Student 15)

As shown in Table 3, most students, however, chose to write their thesis in pairs with a student from the same discipline. At the same time, it was possible for students from different disciplines to generate successful results in the Challenge Lab. Numbers show that students from some disciplines were especially attracted to the Challenge Lab: Civil Engineering (15 students), Environment (10 students), and Technology Management (7 students).

Work across disciplines and with stakeholders

Beyond the learning outcomes listed in the ILOs for all MSc theses, the results from the interviews show that the students perceived that they had developed additional skills that are not offered or usually developed in traditional MSc theses, such as working across disciplines and with stakeholders, as the following two citations show:

Us students representing different educational backgrounds, coming from many different countries and working with stakeholders from industry, academia and government, it took some time and a lot of frustration to understand how to navigate. I think we found a way where we discussed a lot to make sense of, for instance, the meaning of ‘sustainability’ or ‘system’. (Student 7)

Working in a group across educational programmes one gets a broader understanding of how to view a topic or issue from different perspectives. In addition, when having dialogues with different stakeholders the complexity of a challenge becomes evident. Even if one got frustrated, it is also very rewarding to work in this way. (Student 18)

Table 3. The MSc thesis projects, including how close they have come to implementation.

		Students’ discipline background	How far the thesis has come towards reaching implementation^a
Year	Title of MSc thesis project		1	2	3	4	5
2014	1. Back-casting approach to sustainable transport and mobility in Gothenburg. A stakeholders’ perspective on challenges, barriers, and opportunities for sustainability transition	Environment*2	▪	▪
	2. How could a platform for sharing of things be designed? – linking to Chalmers ‘Green Campus’	Environment	▪	▪	▪	▪
	3. Biodiesel in Sweden – barriers, networks and key stakeholders	Physics/Environment	▪	▪
	4. Redefinition of a transport sharing system for a sustainable future	Technology management*2	▪	▪
	5. A pre-study of a potential diffusion of small electric vehicles in the Gothenburg area	Technology management*2	▪
	6. A mobile application for public transport	Civil eng.*2	▪	▪	▪
2015	1. Analysis of perceived insecurities among potential domestic photovoltaic adopters and interface development for a faster diffusion of photovoltaics in Sweden	Energy	▪	▪	▪
	2. Circularity assessment for companies: elements for a general framework	Environment	▪	▪	▪
	3. Interorganisational collaboration in a living lab context, with HSB Living Lab as a case study	Technology management*2	▪
	4. Stakeholders’ requirements and perspectives for future collaboration in solar projects	Technology management	▪
	5. Design of bioretention planters for removal of toxic metals and organic contaminants in stormwater	Environment/Civil eng.	▪
	6. Insights from a benchmarking study of back-casting processes applied to the low carbon transition of West Sweden	Energy/Civil eng.	▪	▪	▪
	7. Positive impact on wellbeing and energy-related behaviours in offices	Civil eng.*2	▪	▪
	8. Criteria of sustainability for new residential buildings	Civil eng.*2	▪
2016	1. How do different factors shape the design of combined goods and waste transportation in urban waterway supply chains?	Maritime eng.	▪
	2. What are the implications of sustainability criteria for urban freight systems in an urban district? – a caste study of Frihamnen	Civil eng.	▪
	3. What sustainability criteria enable guidance in the assessment of the planning proposals for Jubileumsparken in Gothenburg?	Environment*2	▪
	4. What can a mobility service look like in a future DenCity area to act as an alternative to a private car?	Civil eng.*2	▪	▪
	5. How to develop a city strategy for scaling up electro mobility in Gothenburg?	Energy/Environment	▪	▪	▪
	6. How can sustainable renovation of multi-dwellings be ensured from the inventory (pre-design) phase on? – A case study in Gamlestaden	Civil eng.*2	▪	▪
	7. What are the drivers and barriers for implementing innovative sustainable materials into construction projects?	Civil eng.*2	▪	▪
	8. Why is dialogue important in translating sustainability goals into action?	Environment	▪

*2 = two students from the same discipline that did the thesis together.

^a1 = problem formulation; 2 = idea or model generation; 3 = concept development; 4 = testing/evaluation within an academic setting; 5 = testing/evaluation by external stakeholders.

Problem formulation and sustainable development

Three of the ILOs were given mean values >4, i.e. indicating significant learning in all 3 years (Table 2). Two of these ILOs include competences for problem formulation and sustainable development:

• ‘ability to identify, formulate and manage complex problems in a critical, independent and creative manner from an overall perspective’ (ILO No. 4);

• ‘within the framework of the specific degree project, ability to identify which problems need to be addressed to observe sustainable development’ (ILO No. 10).

Below are some citations from students who supported this result.

Learning to frame the question in collaboration with different stakeholders and with a practical relevance has been unique. (Student 2)

Going towards solution too fast is a risk. The difficulty is to understand how to frame the common problem, or challenge, for which to find solutions. This requires the ability to listen and communicate across disciplines. (Student 5)

When I started, I did not realize the difficulty of ‘defining the problem’ across disciplines. It caused me a lot of frustration, but it has been a very valuable experience. (Student 17)

I have done my degree in Industrial Management and Economics. If I would have understood earlier what sustainability is about, I would have shown much more interest in it earlier. Now I will have difficulties because of not working with that perspective. I used to understand sustainability as something that is non-business, non-growth. (Student 11)

Comparing the mean values for these two ILOs with the results from the questionnaire sent in 2016 to all students at Chalmers shows that the corresponding mean values are lower in the questionnaire answers. Especially ILO No. 10 (regarding sustainable development) had a significantly lower mean value, 3.8, compared with the mean value for the Challenge Lab students, which was not below 4.4 in any of the 3 years studied.

Ability to work independently

Another ILO that was rated a mean of >4 in all years is the ‘ability to work independently as a holder of a “Civilingenjör’”, Master of Architecture or Master of Science degree’ (No. 12 in Table 2, my emphasis). The following quotations concern the ability to work independently:

I feel that we have had a great opportunity to use much of the knowledge we have learned from previous studies, in addition to responsibility for creating new knowledge and solutions. I now feel much more confident in my knowledge area. (Student 13)

In Challenge Lab we do not only have the possibility to take responsibility to work independently, but also, the stakeholders expect us to. (Student 18)

How far the master’s theses have come towards reaching implementation

The results of the MSc theses in the Challenge Lab have been assessed in terms of how far they have come towards reaching implementation phase (Table 3). The ambition with Challenge Lab is to allow students to reach beyond the model phase (No. 2), in order to develop their skills related to action competence, by having the project advance into later phases in collaboration with stakeholders. The results indicate that 41% of the students reached the second phase by generating an idea or model, 32% reached the third phase to develop a concept and 23% reached the fourth phase to test/evaluate their project within an academic setting. One thesis project (5%) reached the last phase, of being tested/evaluated by external stakeholders. The interviews show that many students initially had the ambition to reach the evaluation or implementation phase:

When we started our project we first thought we would at least be able to test a solution with stakeholders. But we soon realized how much time collaboration takes, in the different phases, and we only worked on our projects for a semester. (Student 4)

However, as some students pointed out, having an ‘impact’ can also mean that stakeholder awareness is affected as a result of their work and the Challenge Lab process:

When we started, we were told that we would create impact; however, now in the end I cannot see how we could have reached implementation of our project, now that I understand how much time and effort everything takes. But I can clearly see how much learning we have created among us and [among the] stakeholders, and I think our project will be continued by VGR [Region Västra Götaland] also after we have finished our thesis project. (Student 16)

The main result of our work is that both stakeholders we have been working with had not worked together before, but now after this project they see the need and importance of that and therefore will continue to work together in certain areas. (Student 8)

There are examples of projects that have had a particular and strong impact even though they have not neared a tangible implementation result, e.g. project No. 3 in 2015:

The HSB¹ Living Lab project was not going well, but we were told that with our tools and the perspective that we brought in to the work, it really helped the involved stakeholders to come together around common issues and move the project forward. (Student 20)

It should be noted that the goal is to open up possibilities and options, where suitable, for students who are interested in taking steps towards implementation, realisation or commercialisation, but following through to implementation is not a requirement. There are some projects that have been developed and taken further after the students finished their theses, which has been possible because of the interest of stakeholders and the availability of funding. Some examples are projects No. 2 and 6 from 2014, both of which were related to the development of IT software in the context of ‘sharing of things’ and public transportation. Another example is project No. 6 in 2015, whose testing was financed by stakeholders in order to see how ideas for low carbon transition of West Sweden could be further developed.

Discussion and conclusions

In this paper, we have assessed how well some aspects of student learning have been accomplished in the Chalmers Challenge Lab as well as how close the MSc theses done in the Challenge Lab have come to nearing implementation. (For additional aspects of academic learning as well as transformative and integrative values in the Chalmers Challenge Lab, see Holmberg (2014) and Larsson and Holmberg (Forthcoming).) The purpose has been to explore how multiple aims of a CBL environment in higher education can be met without negatively affecting student learning.

We have found that the perceived academic learning of the students who conducted their MSc theses at the Challenge Lab can be considered as normal compared with the ILOs of the MSc thesis at Chalmers. The students’ perceived academic learning in 2016 was largely in agreement with the results from a questionnaire in 2016 sent to all students at Chalmers who performed their MSc thesis in the academic year 2015–2016 and who assessed their own learning. One exception is the perceived learning about sustainable development, which was significantly higher for the Challenge Lab students compared with the Chalmers students in general. This is not surprising as the Challenge Lab focuses on societal challenges for sustainable development.

For a transformation towards sustainable development to take place, competences from different disciplines are needed. Students from different disciplines are therefore invited to take part in Challenge Lab. A potential challenge has been to accomplish specialised knowledge within the students’ main field of study, because the students work around the same theme. The results show large variations among the students. Some students did not perceive that that had gained specialised knowledge within their main field of study. This may be due to the challenge combined with the freedom for students to formulate the research questions.

In addition to the ILOs of MSc theses at Chalmers, the Challenge Lab students perceived that they had gained competence in working across disciplines and with stakeholders, which is in line with the special ambitions of Challenge Lab. Moreover, a previous study indicates that through CBL, the learning outcomes for students at the Challenge Lab are in line with contemporary frameworks for sustainability competences (Hagvall Svensson 2016).

It has not been possible for all thesis projects within the Challenge Lab to go all the way to implementation during the time of the thesis studies. However, implementation is not the only way that the Challenge Lab and its students can create transformative value for society. The students in the present study involved stakeholders in all projects, which may have had some impact, as also shown in some projects. Moreover, an extended opportunity to create impact has been given to some projects that have been funded and continued after the completion of the thesis studies. The aim is to be able to offer more projects of this kind in the future. As many stakeholders have shown interest in collaborating with students in the Challenge Lab, discussions have been initiated with stakeholders on how to arrange for projects to be taken a step further. In Challenge Lab, the role of the students is important. However, the collaborative role of stakeholders is of equal importance in enabling the development of solutions towards a sustainable society.

The integrative value of Challenge Lab, i.e. the role of creating new relations/interaction between the university, companies and the public sector, and the value of seeing Challenge Lab not only as an MSc thesis course, but also (and more importantly) as an arena where different stakeholders’ perspectives on a specific issue are integrated in the work the students do, has been clearly shown to be of great value to the stakeholders as well as to the students: to the stakeholders, as they see the need to have this kind of dialogue and perspective, and to the students, as they gain new perspectives and complementary skills. As an example, in a thesis about electro mobility and sustainable transportation, students created integrative values for stakeholders from the public sector, private sector and academia whom they had invited to a dialogue meeting (Larsson and Holmberg Forthcoming).

From a CDIO perspective, Table 3 shows that CBL experiences similar to those of Challenge Lab can offer opportunities for projects that could strengthen the participants’ skills in problem identification and formulation, multidisciplinary collaboration and communication. Table 3 also shows that it may be demanding and perhaps unrealistic to develop the design, implement and operate skills within the same project, as relatively few projects reach the test/evaluation stages. This could imply that CBL experiences cannot replace more specific design-implement projects in a CDIO programme. However, a programme that features both types of learning experiences will develop a more comprehensive skill set among its graduates.

In conclusion, the results reported in this paper show that it is possible to accomplish multiple aims in a CBL environment in higher education without negatively affecting academic learning. Further, such a learning experience might add the potential for developing skills that are not usually developed in traditional MSc theses.

Acknowledgements

We would like to thank all Challenge Lab students we have interviewed, for lending us their time and engagement. We would also like to thank the anonymous reviewers and colleagues who have given us valuable comments.

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes on contributors

Kamilla Kohn Rådberg is a Researcher at the Department of Technology Management, Chalmers University of Technology, Gothenburg, Sweden. Her research is in the field of innovation management and is related to open innovation. She is also engaged in developing collaborative efforts in research and education.

Ulrika Lundqvist is a Senior Lecturer and Vice Head of Department responsible for undergraduate education at the Department of Space, Earth and Environment at Chalmers University of Technology, Gothenburg, Sweden. Her research interest is Industrial Ecology. She is also engaged in education for sustainable development at Chalmers.

Johan Malmqvist is a Professor in Product Development and Dean of Education at Chalmers University of Technology, Gothenburg, Sweden. His current research focuses on information management in the product development process/product lifecycle management and on curriculum development methodology.

Oskar Hagvall Svensson is a Ph.D. candidate in Entrepreneurship and Strategy at the Department of Technology Management and Economics at Chalmers University of Technology, Gothenburg, Sweden. His research is focused on entrepreneurial learning in engineering education.

Notes

1 HSB is a cooperative housing association where the tenants are owners and members: https://www.hsb.se/kil/om-hsb/in-english/. This association has established a living lab: https://www.hsb.se/hsblivinglab/Om/.