Educating collaborative planners: strengthening evidence for the learning potential of multi-stakeholder regional learning environments

Abstract

Planning education needs effective learning environments that support students’ boundary crossing competence development. The multi-stakeholder regional learning environment (RLE) is, by its typical design, hypothesized to foster boundary crossing. This quasi-experimental mixed method pre- and post-test study affirms the effectiveness of the RLE for stimulating competence development, and specifically shows the added value of three typical ‘boundary crossing’ RLE design characteristics, i.e. working in multidisciplinary student groups, working intensively with multiple stakeholders, and a high coaching intensity. Results strengthen previous findings, and contribute to the debate in planning education on an evidence-based pedagogical fundament for commonly used authentic learning environments.

Keywords:

• Planning education
• student learning
• authentic learning environment
• multi-stakeholder
• boundary crossing

Introduction

Planning education needs effective learning environments that encourage students to cross the boundaries of disciplines and practices in preparation for a collaborative, ‘boundary crossing’ planning profession. The regional learning environment (RLE) has recently been established in various Dutch regions by multiple planning actors, including education and research, to work on sustainable regional development. The RLE is expected to stimulate boundary crossing learning. In the RLE, students work in groups on transdisciplinary (Scholz & Steiner, 2015), regional planning problems identified by and to be solved with actors in the field. Solving the transdisciplinary problems requires co-creation of new knowledge between students on the one hand, and researchers, policy-makers, members of NGOs, entrepreneurs and/or citizens on the other hand. The end result is meant to be of value for the external problem holder and to contribute to regional development. The RLE provides students with the opportunity to ‘cross boundaries’ between multiple disciplines and perspectives and learn from that (Akkerman & Bakker, 2011). As such, the RLE allows students to develop their abilities for working across boundaries (Walker & Nocon, 2007).

Evidence for the effectiveness of this new learning environment for student learning is needed to fund its reason for existence and strengthen its further development (Slavin, 2008). Confirming the effectiveness aligns with current debates in planning education that stress the importance of more in-depth pedagogical understanding of what works in learning environments as part of planning curricula (Angotti et al., 2011; Long, 2012a, 2012b; Frank et al., 2014). A previous study (Oonk et al., 2013) examined the effectiveness of the RLE for planning students’ learning, and the added value of working in multidisciplinary student groups, with multiple stakeholders and with a high coaching intensity. These three learning environment characteristics were expected to stimulate learning across boundaries, and as such foster students’ boundary crossing competence development. Overall, the five studied RLEs, in which a total of 225 students participated, resulted in significant learning gains for students. This previous study also showed that both working in multi- versus monodisciplinary student groups, and with a high versus a low level of coaching intensity, fostered learning. However, the expected positive learning effects of working intensively with multiple stakeholders were not found.

This previous result triggered further investigation since both planning theory and practice stress the undeniable importance of multi-stakeholder collaboration for the planning profession (e.g. Dalton, 2007; Sletto, 2010; Edwards & Bates, 2011). Encouraging students to work in close collaboration with multiple stakeholders is expected to stimulate their development of boundary crossing competence (Wenger, 2000; Walker & Nocon, 2007; Oonk et al., 2013). To strengthen evidence for the RLEs’ effectiveness and further explore the partly surprising results of the previous study with respect to learning with and from multiple stakeholders, this follow-up study examined student learning in seven other RLEs in which a total of 143 students participated.

On the basis of the combined results of the current study and the previous study (Oonk et al., 2013), we discuss the effectiveness of the RLE for planning students’ learning in relation to its three boundary crossing design characteristics. With regard to identified similarities between the RLE and other authentic learning environments as used in planning education e.g. the planning studio and service learning environments (Oonk et al., 2013), results of the two studies contribute to the call for evidence-based pedagogical and didactical improvement of authentic learning environments as used in planning education worldwide (Angotti et al., 2011; Long, 2012a, 2012b).

This paper starts with positioning the planning profession in a boundary crossing perspective. Secondly, the educational design of the RLE will be explained, including its typical ‘boundary crossing’ learning environment characteristics (Akkerman & Bakker, 2011) that are supposed to stimulate nowadays planning competencies. These two descriptive sections introduce the accounts for the research questions and design of this follow-up study.

Theoretical framework

Boundary crossing as a key capability of current planners

The current planning profession requires professional planners to set up, facilitate and/or act in complex collaborative planning processes in which they collaborate with multiple stakeholders from diverse disciplinary backgrounds representing a diversity of interests (Healey, 1997, 2003; Forester, 1999; Innes & Booher, 1999, 2004; Allmendinger, 2009). As such, planning professionals ‘cross boundaries’ (Akkerman & Bakker, 2011, p. 133). Boundary crossing at this place is a key concept for describing the ‘efforts by individuals or groups at boundaries to establish or restore continuity in action or interaction across practices’ (Bakker & Akkerman, 2014, p. 225). In a boundary crossing process we acknowledge that one person cannot be an expert in all sites, and that solving complex problems requires the collaborative creation of new knowledge across sites around the boundary, explicitly addressing differences between sites (Akkerman & Bakker, 2011). Boundary crossing competence is then described as ‘the ability to function competently in multiple contexts’ (Walker & Nocon, 2007, p. 178), and manage, switch between, and integrate multiple discourses and practices across social boundaries (Lansu et al., 2013).

Many scholars from the fields of planning, transdisciplinary sustainability research, communication and education have investigated professional tasks and competencies required for current professional planners, both from a theoretical and/or a practical perspective (e.g. Alexander, 2001; Guzzetta & Bollens, 2003; Dalton, 2007). Two key professional tasks and required competencies to perform these tasks (Mulder &Winterton, in press) can be extracted from relevant scholarly resources on this topic. Both tasks and related competencies carry elements of boundary crossing and boundary crossing competence, acknowledging that multi-stakeholder processes in today’s planning aim at co-creating new knowledge across disciplinary professional boundaries.

The first key task is to set up multi-stakeholder processes. To carry out this task, planners should be able to understand the field of actors and their interests (Schön, 1983; Healey, 1998, 2003), and the planning process in which these multiple actors are involved (Healey, 1998, 2003; Seltzer & Ozawa, 2002). Next, planners should be able to cultivate community networks (Forester, 1989; Greenlee et al., 2015) and organize effective organizational structures that promote collaboration (Alexander, 2009; Balassiano, 2011).

The second key task is to act in and facilitate multi-stakeholder processes. Competencies required to carry out this task mainly address communicative capabilities to be able to facilitate respectful and effective discussions within and between communities. Discussions in these processes are characterized by differing, sometimes competing, and in time varying needs, intentions, values, norms and beliefs, and inequities in power (Healey, 1993). Facilitating planners should strategically approach the ‘governance of place’ (Healey, 2003, p. 116), paying attention to both the qualities of place and process, embrace what is new and experimental, and anticipate political and economic pressure (Forester, 1989; Booher & Innes, 2002; Higgins et al., 2009). Participating in and/or facilitating these discussions requires the competencies to understand and interweave knowledge of different disciplines (Umemoto, 2001; Seltzer & Ozawa, 2002; Akkerman & Bakker, 2011; Opdam et al., 2015); to listen carefully (Forester, 1989, 1999), to exchange values and beliefs, and use criteria for ideal speech (e.g. Habermas, 1984). While discussing, planners should be able to see multiple perspectives, to admit to differences, to enable others to communicate authentically, and to acknowledge different arguments in a plan (Healey, 1993; Booher & Innes, 2002; Seltzer & Ozawa, 2002; Higgins et al., 2009). Wiek et al. (2011) add at this place the ability to help others switching quickly between scale levels in space and time. Talking about the decision-making phase in complex multi-actor processes, Balassiano (2011) mentions the ability to facilitate reflection that encourages regular and systematic evaluations of efforts and mutual learning. This would enable to make legitimate decisions that reflect a comprehensive understanding of values and issues and improve equity. To stimulate clarity and openness of planning processes, planners should also be able to write informative, engaging short pieces for the general public (Seltzer & Ozawa, 2002; Greenlee et al., 2015).

Boundary crossing in planning education

To prepare planning students to work in a collaborative boundary crossing context, planning education needs learning environments that support students to develop their boundary crossing competence (Booher & Innes, 2002; Bourner, 2010; Edwards & Bates, 2011). These learning environments should include learning elements that challenge students to adopt the above described tasks and competencies. That is, the learning environments should provide students optimal opportunities for learning ‘at the boundary’ (Akkerman, 2011; Angotti et al., 2011; Balassiano, 2011). In planning education, different authentic learning environments have been designed to stimulate planning students’ boundary crossing tasks and competencies, such as the planning studio (Higgins et al., 2009; Balassiano & West, 2012; Long, 2012b) and service learning (Roakes & Norris-Tirrell, 2000; Sletto, 2010; Angotti et al., 2011). These learning environments comprise, to differing degrees but not always, real world planning problems, multi-disciplinary group work, involvement of external stakeholders and critical reflection on the learning experiences (Oonk et al., 2013).

Experiences of students and educators working in these authentic learning environments as used in planning education have broadly been described by scholars over the world, addressing a broad variety of design characteristics of these learning environments (Ward, 1999; Giles et al., 2001; Sletto, 2010; Long, 2012b). However, research on innovations in the design of these learning environments, e.g. on the impact of student-stakeholder collaboration, rarely articulates the educational theory that informs their design (Billig & Eyler, 2003; Angotti et al., 2011; Long, 2012a; Nemeth & Long, 2012). Consequently, the educational effectiveness of design innovations in terms of student learning outcomes has hardly been assessed, certainly not in terms of boundary crossing competence. This study, in combination with its predecessor, aims at more systematically confirming evidence for the effectiveness of an exemplary multi-stakeholder learning environment, i.e. the RLE, with its typical boundary crossing design characteristics, for planning students’ learning. As such, this study starts to fill the evidence-based gap in contemporary planning education, and advocates the importance of explicitly including boundary crossing elements in authentic learning environments.

The Regional Learning Environment and its boundary crossing learning potential

Since 2005, various Dutch academic and vocational institutes for life science education participate in the multi-stakeholder RLE. The ultimate aim of the RLE is to collaboratively create new knowledge and expertise amongst participants from society and educational institutions that supports sustainable regional development and/or transition. From an educational perspective, the RLE is an authentic (compare e.g. Newmann & Wehlage, 1993; Herrington & Oliver, 2000; de Kock et al., 2004), demand driven, collaborative learning environment providing rich opportunities for students from different study programmes, including planning students, to work in a boundary crossing context by explicitly addressing various disciplines and multi-stakeholder collaboration (Meijles & Van Hoven, 2010; Oonk et al., 2013; Foorthuis et al., 2012). Project assignments in most cases originate from a long term regional knowledge agenda set for the regional development process. These agendas are the result of a continuing collaborative effort of multiple stakeholders all having an interest in the future development of a particular region. Figure 1 illustratively describes working processes, examples of executed projects, and project deliverables in the RLE Salland, being representative for the RLEs included in this study. The next paragraph explicates the educational design characteristics of the RLE.

Figure 1. Illustration of the state of affairs in the RLE Salland.

The authentic RLE is always characterized by the following learning environment design characteristics. Students are exposed to a professional experience by introducing them to real-world problems in a real-world situation. They work on a transdisciplinary assignment from a real external problem holder (Scholz & Steiner, 2015). Assignments concern regional planning problems identified by actors (persons or organizations) in the field. Working on the assignments always engages students in authentic, wicked tasks and activities.

Students work in groups as solving a transdisciplinary, complex problem with various possible solutions requires collaborative knowledge construction. Collaborative knowledge construction should involve the integration and co-creation of knowledge or in Watson’s social constructivists words the sharing or building of meaning via reflection on and articulation of different ideas and perspectives (Watson, 2001).

Working in the RLE results in a realistic authentic product that potentially has value for the external problem holder(s) and contributes to regional development and/or transition. This product is mostly presented to the external problem holder(s) in a final presentation. In practice, deliverables vary as a result of agreements between the students, their teachers and the external problem holder(s) (see Figure 1 for some examples of products). Next to the product for the external problem holder(s), process reflection reports, either on an individual or group level, are often required as a deliverable. Product and process deliverables are both part of the assessment, but assessment criteria and procedures vary between RLEs.

The teacher’s role is to facilitate and/or coach the learning process instead of transferring knowledge as an expert. Additionally, the teacher is also a learner, working in an almost equal relationship with the students to collaboratively tackle complex regional problems. In some cases, the teacher is responsible for the quality assurance of the final product towards the external problem holder(s).

From a boundary crossing perspective, the RLE preferably has three additional design characteristics (see also Sletto, 2010; Wesselink et al., 2011; Foorthuis et al., 2012; Oonk et al., 2013):

(1)	students work in multidisciplinary student groups, which means that the groups consist of students from different study programs, i.e. disciplines;
(2)	students intensively collaborate with multiple stakeholders like researchers, policy-makers, members of NGO’s, entrepreneurs and/or citizens. Multi-stakeholder collaboration is needed to solve a transdisciplinary, regional problem with an unknown answer. All parties involved have different stakes in, and perspectives on the issue at hand;
(3)	students are intensively coached on explicating learning across boundaries. Teachers structurally stimulate and support students to jointly reflect on working across the boundaries of disciplines and perspectives, and explicate learning thereof.

Finding evidence for planning students’ learning in the RLE

Dutch planning schools practice the RLE since 2005. From the beginning, RLEs delivered useful products, and students, teachers and multiple stakeholders reported valuable learning results from working in the RLE. However, a systematic investigation of the effectiveness of the RLE for planning students’ learning is lacking hitherto. In search for more evidence-based educational practices, empirical evidence for the RLEs’ effectiveness is essential (Slavin, 2008). To build evidence for student learning in the RLE related to its learning environment characteristics, and more specifically for the effect of the typical boundary crossing learning environment characteristics, a previous quasi-experimental mixed method study has been carried out (Oonk et al., 2013). This previous study examined 225 students (N = 225) participating in five RLEs, and either working in mono- or multidisciplinary student groups, with a low or a high stakeholder collaborative intensity, and with a low or a high coaching intensity. The previous study than compared planning students’ learning in terms of competence development (quantitative part) and reported other learning outcomes (qualitative part) in five RLEs in which the students differed in the extent to which they worked with these three typical boundary crossing learning environment characteristics. Results showed significant growth of professional expertise and several generic competencies in four out of the five studied RLEs (Table 1). Next, the expected effects of working in multidisciplinary groups and with a high coaching level were found, while effects of intense multi-stakeholder collaboration were not revealed, neither in quantitative nor in qualitative data. Contemplating this unexpected result in the light of the crucial importance of multi-stakeholder collaboration for planning, and the expectation that this collaboration could optimally be trained in a real-life multi-stakeholder learning environment, triggered a further examination of student learning in the RLE. We decided to carry out a follow-up study to further explore and strengthen evidence for students’ learning in the RLE, and the added value of typical boundary crossing characteristics.

Table 1. Characteristics of and developed competencies including effect size in five previously studied RLEs (Oonk et al., 2013).

	RLE 1.	RLE 2.	RLE 3.	RLE 4.	RLE 5.
General characteristics
Number of students (n)	81	15	52	64	13
Educational level	AcademicYear 4 or 5	Professional Higher EducationYear 3	AcademicYear 1	AcademicYear 2	Professional Higher EducationYear 3
Study load	8 weeks fulltime	16 weeks2 days/week	2 weeks fulltime	8 weeks1 day/week	20 weeks2 days/week
Student group size	5 – 7, mostly 6	6 or 9	6	5 – 6	3 – 4
Students’ mean age	25	22	20	21	20
Students’ gender[# (%)]	♂ 31 (38%)♀ 50 (62%)	♂ 10 (67%)♀ 5 (33%)	♂ 30 (58%)♀ 22 (42%)	♂ 51 (80%)♀ 13 (20%)	♂ 9 (69%)♀ 4 (31%)
Classification
Mono-/multi-disciplinary groups	Multi	Multi	Mono	Mono	Mono
Active stakeholder collaboration	Yes	Yes	Yes	No	No
Coaching intensity	High	High	Low	Low	Low
Results
Amount and type of developed competencies as a result of paired sample t-tests^a	All 9 competencies	5 out of 9 competenciesDomain-specific professional expertise Deciding and initiating activities Collaborating and discussing Planning and organizing Acting customer oriented	4 out of 9 competenciesDomain-specific professional expertise Creating and innovating Collaborating and discussing Investigating	3 out of 9 competenciesDomain-specific professional expertise Collaborating and discussing Acting commercially	None
Effect size of competence development (in Cohen’s d^b )	0.3 – 0.67medium	0.69 – 1.29medium to large	0.32 – 0.97medium to large	0.39 – 0.64medium	Not applicable

^a p < 0.05.

^b d < 0.2 = small; d ± 0.5 = medium; d > 0.8 = large.

This follow-up study examines seven other RLEs. The study firstly re-investigates if working in a RLE stimulates the development of planning students’ professional expertise and generic competence (hypothesis 1). Next, it assumes that planning students’ competence development in the RLE is strengthened by the learning environment characteristic of ‘working in multi-disciplinary student groups’ (hypothesis 2), ‘a high level of multi-stakeholder collaboration’ (hypothesis 3) and ‘a high coaching intensity’ (hypothesis 4). The research questions for this follow-up study are to what extent these hypotheses can be supported, and to what extent the findings differ from the findings of the previous study. In other words, do we reaffirm competence development across RLEs? And next, do we now find a differential effect of all three typical ‘boundary crossing’ learning environment characteristics on student learning? Results of the study will show further evidence for the added value of the RLE and its typical learning environment characteristics.

Methodology

This study was designed, similar to its predecessor (Oonk et al., 2013), in a quasi-experimental mixed-method pre- and post-test design investigating student learning in a sample of seven RLEs. The method section consecutively describes the RLEs and its participants, the data sources, and the way in which the data have been analyzed.

The RLE and its participants

Seven RLEs, as implemented in different planning education programs, were monitored: two in academic study programs (n = 23; 13) and five in professional higher education programs (n = 33; 25; 22; 14; 13). To make this selection, teachers from eight different planning programs who actively participate in RLEs were contacted at the start of the study (Summer, 2012). Seven out of eight contacted teachers agreed to participate in the study with a specific RLE project. The selection finally included five RLEs running in Dutch planning schools during the academic year 2012–2013 (see Table 2, RLE 1–5) and two during the first semester of the academic year 2013–2014 (see Table 2, RLE 6 and 7). To guarantee anonymity, the RLEs are identified by number. Table 2 shows the characteristics of the participating students (N = 143) in the seven monitored RLEs.

Table 2. Characteristics of and developed competencies including effect size in the studied RLEs.

	RLE 1.	RLE 2.	RLE 3.	RLE 4.	RLE 5.	RLE 6.	RLE 7.
General characteristics
Number of students (n)	23	22	13	33	13	25	14
Educational level	AcademicYear 4 or 5	Professional Higher EducationYear 3	Professional Higher EducationYear 3	Professional Higher EducationYear 3	AcademicYear 3	Professional Higher EducationYear 3	Professional Higher EducationYear 3
Study load	8 weeks fulltime	20 weekshalf-time	20 weekshalf-time	2 weeksfulltime	5 weeksfulltime	20 weekshalf-time	20 weekshalf-time
Student group size	5 – 7, mostly 6	4 – 5	4 – 5	6	7	3 – 5	3 – 4
Students’ mean age	24	22	23	23	20	21	22
Students’ gender[# (%)]	♂ 8 (35%)♀ 15 (65%)	♂ 20 (91%)♀ 2 (9%)	♂ 11(85%)♀ 2 (15%)	♂ 18 (55%)♀ 15 (45%)	♂ 8 (62%)♀ 5 (38%)	♂ 18 (72%)♀ 7 (28%)	♂ 4 (29%)♀ 10 (71%)
Classification
Mono-/multi-disciplinary groups	Multi	Multi	Multi	Multi	Mono	Multi	Mono
Active stakeholder collaboration	Yes	Yes	Yes	No	No	Yes	Yes
Coaching intensity	High	Low	High	Low	Low	Low	Low
Results
Developed competencies as a result of paired sample t-tests^a	All competencies except for Acting customer oriented	All competencies except for Planning and organizing andBuilding relationships and networking	All competencies	3 out of 10 competenciesDomain-specific professional expertiseDeciding and initiating activitiesPlanning and organizing	All competencies except for Collaborating and discussing andActing customer oriented	2 out of 10competenciesCollaborating and discussingInvestigatingActing commercially	None
Effect size of competence development (in Cohen’s d^b )	05 – 1.2medium to large	0.62 – 1.21medium to large	0.8 – 1.79large except for 1x d = 0.65	0.39 – 0.48medium	0.61 – 1.24medium to large	0.47 – 0.57medium	Not applicable

^a p < 0.05.

^b d < 0.2 = small; d ± 0.5 = medium; d > 0.8 = large.

To check if the studied RLEs met the general RLE learning environment characteristics (see section The Regional Learning Environment and its Boundary Crossing Learning Potential) various data sources were triangulated. Observations, interviews with teachers, document analyses and students’ posttest scores (as described below) on propositions regarding the learning environment design characteristics of the RLE, confirmed that all studied RLEs met these RLE design characteristics. The RLEs differed regarding educational level, study load, size of the student groups and total number of students involved (Table 2).

RLEs also differed in the extent to which they fulfilled the boundary crossing learning environment characteristics. Students in every RLE were classified as working (1) in a mono- or multidisciplinary student group, (2) at a low or a high level of multi-stakeholder collaboration and (3) with a low or a high degree of coaching. In mono-disciplinary student groups only planning students participated. In multidisciplinary student groups planning students collaborated with students from other study programs (e.g. landscape architecture, environmental sciences, forestry and nature conservation, and management studies). For the purpose of classifying students as having worked at a low or a high level of stakeholder collaboration, students were asked several questions in a post-test questionnaire regarding their collaboration with various external stakeholders. A student was classified as working at a low level of multi-stakeholder collaboration if the student only read information about the stakeholders and their opinions without contacting them personally, or asked the stakeholders informative questions that were answered without any discussion. A student was classified as working at a high level of multi-stakeholder collaboration if the student discussed project-related issues with multiple stakeholders or really worked together in collaborative working sessions with multiple stakeholders during the project. Students were classified as working with a low degree of coaching when they only met their teacher/coach once or twice a week for a group discussion mainly focused on the progress towards a final result. Students in RLEs with a high degree of coaching followed an intensive parallel coaching trajectory in which learning experiences across the boundary were explicitly addressed and utilized to optimize the RLE product and process. Based on the majority of individual student classifications for a boundary crossing learning environment characteristic in a RLE, every RLE was generally classified for all three boundary crossing characteristics (Table 2).

Data sources

A validated pre- and post-test questionnaire (Bartram, 2011; COLO, 2006; Khaled et al., 2014) assessed students’ perceived level of domain specific professional expertise and nine generic competencies (Table 3). Nine of this total of 10 competencies were identified by 56 planning students and 10 RLE teachers as crucial and typical in the context of RLEs and used in a pre- and post-test questionnaire in the previous study (Oonk et al., 2013). Remarkably, this selection did not include the competency ‘Building relationships and networking’. To make sure that we collected as many as possible indicators for the effect of multi-stakeholder collaboration on student learning, after the unexpected absence of it in the first study, we decided to add the competency ‘Building relationships and networking’ to the questionnaire used in this follow-up study.

Table 3. Competencies including description as assessed in pre- and post-test.

No.	Competency and competency description
1	Domain specific professional expertise
	I have much expert knowledge and professional skills. By applying this knowledge and these skills, I can easily perform standard operations in my field of expertise. I also have enough expert knowledge and skills to be able to act in a professional way in new and unexpected situations. With my expertise I am able to help others to complete their tasks.
2	Deciding and initiating activities
	In my work or my studies, I take the initiative to undertake activities and start up and tackle (new) tasks. When taking a decision, I first think about the options I have and which advantages and disadvantages these options have. Because of this, I can explain others why I made certain choices and/or I can justify my choices. This also makes me perform my tasks with confidence, and I take responsibility for my choices.
3	Showing attention and understanding
	I can empathize with other people’s feelings and take their opinions into account. I do this by carefully listening, paying attention to signals of others and showing understanding for the views of others, even if they differ from my own views. I adapt my reaction and behavior to the situation that I am in, as well as to the feelings of others.
4	Collaborating and discussing
	When working with others, I give an active and valuable contribution to our meeting. I perform my duties and do my best to work with the group towards a good result. Wherever possible, I help my group members with their duties and give them feedback on their work. I also contribute to a good atmosphere in the group. I am aware of irritations between people and take action to prevent and resolve conflicts.
5	Investigating
	I have a research-oriented, curious attitude. If something is unclear to me, if I want to develop something new or if I encounter a problem, I start looking for solutions. I collect information in a structured way from various sources such as internet, journals and experts. Based on this information I can identify several solutions and I test whether solutions work appropriately. Based on my research, I formulate recommendations and I share my findings with others in order to let them benefit from my findings.
6	Acting commercially
	I understand business processes. This means that I have knowledge about the policies, the goals and the way a company wants to achieve those goals, also called the business strategy. I also know how the different sections/departments of a company are related. I know how to find the right people for the right tasks and in case of questions I know whom to contact. Acting commercially also requires operating cost effectively. Before spending money, I weigh the costs and benefits for the company. I also minimize unnecessary costs and a waste of energy and materials.
7	Creating and innovating
	I want to be innovative in my work and I regularly come up with new ideas for my work. I am concerned with the improvement of products and/or services or with inventing new things. When doing this, I take into account current developments in my professional field, such as new trends, needs of society, and needs of future customers. I am capable of assessing whether my new ideas are feasible. I discuss my ideas with others, for example my colleagues, and take actions to actually execute my ideas.
8	Planning and organizing
	When I’m performing a task, I make a plan in advance. I first consider which results I want to achieve, which tasks must be performed to achieve the results, how much time is required and which materials I need. During the execution I monitor if everything runs according to the time schedule. If not, I adjust the time schedule.
9	Customer-oriented acting
	When working for a client or customer, I make sure that I know the needs of my client/customer. I adapt my work to these needs. I have a regular contact with my clients/customers to check their satisfaction, and keep them informed about the progress. I can answer questions of clients and customers. In case of a complaint, I quickly undertake action to resolve the complaint.
10	Building relationships and networking
	I think contacting other people in my work is important. I take initiative to contact people that, now or in the future, could positively contribute to my work. I easily get in contact with people within my organization, and also with people outside my organization. I do maintain my professional contacts, to get in touch with other people if we can do something for each other.

The questionnaire consisted of a description of each of the 10 competencies (Table 3) and 4–6 performance indicators per competency (e.g. ‘When working with others, I actively contribute to our meeting’ (for ‘Collaborating and discussing’) or ‘I show understanding for other peoples’ views’ (for ‘Showing attention and understanding’). In both pre- and post-tests students awarded themselves a score for each performance indicator on a 10-point scale. A competence mean score was based on students’ rating of the 4–6 performance indicators per competency. At the start of the project, directly after being informed about their project assignments, students filled out the pre-test. At the end of the project, right after the final presentation of the project result, they filled out the post-test. The scales were reliable (∝ > 0.80), except for one scale. The non-reliable scale became reliable (∝ > 0.80) after the deletion of one item. RLEs were compared on their development scores between pre- and post-test (dependent variables).

To enrich the quantitative data on student learning with qualitative data, the post-test asked students to answer the open question ‘What did you learn more from your RLE project? Please write down as many of your ideas as possible regarding learning in this project’. The answers provided additional insights into students’ ‘other learning outcomes’.

Analysis

Paired sample t-tests were used to calculate development of the students per RLE on the 10 competencies, comparing pre- and post-test scores on the competency scales. Effect size for the paired sample t-tests was measured in Cohen’s d with d < 0.2 showing a small effect, d around 0.5 showing a medium effect and d > 0.8 showing a large effect (Cohen, 1988).

Three multivariate general linear models (GLMs) compared competence development as a function of the three boundary crossing learning environment characteristics using mono-/multidisciplinary groups, low/high multi-stakeholder collaboration, and low/high coaching intensity as independent variables. Effect size for the GLMs was measured in partial Eta-squared (partial η²) with partial η² ≈ 0.01 showing a small effect, partial η² ≈ 0.06 showing a medium effect and partial η² ≈ 0.14 showing a large effect (Cohen, 1988).

The reported learning outcomes from the students were top-down coded on ‘referring to working in multidisciplinary student groups’, ‘referring to collaborating with multiple stakeholders’ or ‘referring to other learning environment characteristics’ (Miles & Huberman, 1994). We did not analyse the students’ reported learning outcomes on the third independent variable ‘coaching intensity’, since teacher coaching was not expected to be explicitly reflected in the student reported learning outcomes. The coding work was partly carried out by the first two authors independently in order to determine an interrater reliability score (Cohen’s Kappa (к)). This score (к) was 0.90 which represents an almost perfect strength of agreement (Landis & Koch, 1977). Coding differences between the two raters were discussed until agreement, after which the first author accomplished the remaining coding work. After coding, the percentages per code were calculated for students who worked in either mono- or multidisciplinary student groups or with a low or a high level of stakeholder collaboration. This was done to examine if these different groups reported different learning outcomes in relation to these boundary crossing learning environment characteristics. To illustrate students’ reports, two representative examples of learning outcomes per category were chosen. Finally, the findings of these analyses were compared to those of the previous study.

Results

Results of this follow-up study

Results confirmed hypothesis 1 by showing significant competence development in six out of seven RLEs (see Table 2).

Competence development ranged from no significant development (RLE 7), via few developed competencies (# = 3 in RLE 4 and 6), to significant development of (almost) all competencies (RLE 2 (# = 8), RLE 5 (# = 8), RLE 1 (# = 9) and RLE 3 (# = 10)). Effect size of the competence development ranged from medium to large.

The RLEs showed differences with respect to which competencies they stimulated. ‘Domain specific professional expertise’ and the generic competencies ‘Deciding and initiating activities’, ‘Investigating’ and ‘Acting commercially’ developed in almost all RLEs. The competency ‘Building relationships and networking’, added to the test of the previous first study, developed in three RLEs. Results did not show differences in competence development per RLE for gender and age.

Hypotheses 2, 3 and 4 were all confirmed by GLM analyses showing a large (partial η² ≥ 0.14) positive multivariate effect of all three independent boundary crossing learning environment characteristics on competence development, that is of:

(1)	multidisciplinary student groups (F(10, 111) = 2.736, p < 0.05, partial η² = 0.198);
(2)	a high level of multi-stakeholder collaboration (F(10, 111) = 1.975), p < 0.05, partial η² = 0.151); and
(3)	a high coaching intensity (F(10, 111) = 2.339, p < 0.05, partial η² = 0.174).

This means that, measured across the seven RLEs, all three boundary crossing learning environment design characteristics more strongly stimulate students’ competence development than their antagonists. These findings were strengthened by the results of the qualitative analysis of students’ reported learning outcomes related to the boundary crossing learning environment characteristics. Students working in multidisciplinary student groups referred in 18% of the reported learning outcomes to learning from multidisciplinary group work. Students working in mono-disciplinary student groups did so in 5% of the reported learning outcomes. Students working at a high level of multi-stakeholder collaboration referred in 38% of the reported learning outcomes to learning from multi-stakeholder collaboration. Students working at a low level of multi-stakeholder collaboration did so in 10% of the reported other learning outcomes (see Table 4 for percentages and representative examples of reported learning outcomes).

Table 4. Percentages of students’ reported ‘other learning outcomes’ referring to multidisciplinary group work and multi-stakeholder collaboration per category of learning environment characteristics.

Categories	# Students per category	# Excerpts per category of students	# (%) of Excerpts that refers to multidisciplinary group work	Illustrative examples
Students working in multidisciplinary student groups	116	231	41 (18%)	I learned how to better collaborate with people from different disciplines.I learned to rely on the skills of group mates.
Students working in mono-disciplinary student groups	27	66	3 (5%)	I learned how to communicate with people from different backgrounds.Just make sure you have the right knowledge and skills to adapt to each situation.
TOTAL # students	143	297	-
Categories	# Students per category	# Excerpts per category of students	# (%) of Excerpts that refers to multi-stakeholder collaboration	Illustrative examples
Students working on a high level of multi-stakeholder collaboration	97	199	75 (38%)	It is important to maintain contacts with the client and stakeholders to prevent for misunderstanding.I learned how to deal with the criticism you have to withstand during and after a community discussion meeting.
Students working on a low level of multi-stakeholder collaboration	46	98	10 (10%)	My task was to contact the stakeholders. I am no longer afraid to call to complete strangers.We learned that working for a real external client is serious business.
TOTAL # students	143	297	-

Comparing the results of the two successive studies

A comparison between the results of the previous study 1 (Table 1) and this follow-up study 2 (Table 2) showed the following. In both studies all RLEs, except one, showed competence development. The average amount of significantly developed competencies over the RLEs was 44% in study 1 and 59% in study 2. In both studies ‘Domain specific professional expertise’ developed in almost all RLEs. With respect to generic competence development, almost all RLEs in study 1 developed the competence ‘Collaborating and discussing’, while study 2 found the competencies ‘Deciding and initiating activities’, ‘Investigating’ and ‘Acting commercially’ to develop in almost all RLEs. Additionally, the overall effect size of the competence development was higher in study 2.

Where study 1 only found a positive effect of working in multi-disciplinary student groups and high coaching intensity on student learning, study 2 corroborated all three independent variable effects including a large positive effect of a high level of multi-stakeholder collaboration both in quantitative and in qualitative data.

Conclusion

This study confirms the hypothesis that the RLE stimulates planning students’ competence development. As such, the study provides evidence for the effectiveness of RLEs in the sense of developing both domain specific professional expertise and generic competence of planning students. Although the pattern of competence development differs per RLE, six out of seven studied RLEs show significant competence development. The competency ‘Building relationships and networking’ significantly developed in three out of seven RLEs (Table 2). The average amount of developed competencies per RLE and effect sizes for development are higher than in a similar previous study (Oonk et al., 2013; see Table 1). This reconfirms and strengthens findings from our previous study that found competence growth in four out of five RLEs.

The three identified typical ‘boundary crossing’ learning environment characteristics of working in multidisciplinary groups, a high level of multi-stakeholder collaboration, and a high coaching intensity had a large positive effect on competence development. This confirms hypothesis 2–4, meaning that all three boundary crossing characteristics stimulate student learning in the studied RLEs. Qualitative data on students’ reported learning outcomes as a result of working in multi-disciplinary groups and with multiple stakeholders corroborated the effects of these learning environment characteristics. Findings on the positive effects of the three ‘boundary crossing’ learning environment characteristics add to the findings of the previous study, in which only multi-disciplinary student groups and a high coaching level showed a positive multivariate effect.

Repeated and additional evidence for student learning in the RLE exposes the potential of the RLE to prepare students for their key professional tasks in setting up and facilitating multi-stakeholder processes for which they should be able to work across the boundaries of multiple disciplines, practices and perspectives of a variety of stakeholders involved (Booher & Innes, 2002; Bourner, 2010; Edwards & Bates, 2011).

Discussion

The confirmation of student learning in the RLE is important on planning educators’ way to further develop effective boundary crossing learning environments for future planners. More specifically, the fact that both domain specific professional expertise and various generic competencies are stimulated in RLEs, even more in the presence of various boundary crossing learning environment characteristics, supports the added value of RLEs for planning education. These findings also challenge the design of, often used, other authentic learning environments in planning education, like studios or service learning, to more explicitly incorporate these boundary crossing learning environment characteristics.

The fact that the amount of developed competencies and the effect size of development is higher in the second follow-up study, in comparison to a previous study as carried out a few years earlier (Oonk et al., 2013), could indicate that slight educational improvements of the RLE over time sort their effect. Our observations of the studied RLEs suggest that the design and implementation of the RLEs have improved in sense of e.g. clarity in assignments and working processes, and professional development of teachers in their coaching role. This implies that paying careful attention to the instructional design of RLEs, their implementation, and the role teachers play in these environments, can affect the amount and type of competencies to be developed. Future research should support the further instructional design of the RLE, and the implementation of its typical boundary crossing learning environment characteristics, by obtaining more in depth insight into the actual teaching and learning processes in the RLE.

Both studies strongly confirm the added value of working in multi-disciplinary student groups for student learning in the RLE. This finding argues for an attempt to work as much as possible in multi-disciplinary groups in authentic learning environments, although this is often hard to organize in higher education settings. We even pose an argument for a duple effect of this learning environment characteristic on students’ boundary crossing competence. Students first get the opportunity to cross boundaries of disciplines in their relatively safe in-school group environment, and together prepare for boundary crossing in the out-of-school transdisciplinary settings in which they will participate.

In contrast to the first study, this study also shows a large positive effect on competence development of intense multi-stakeholder collaboration, confirming our initial expectations. Unexpectedly, the competence ‘Building relationships and networking’ developed in only three RLEs while we would expect this to develop in all RLEs, and certainly in the RLEs with a high level of multi-stakeholder collaboration. Our explanation for this finding is that students do not optimally make use of learning from, and together with, multiple stakeholders. This explanation is prompted by the fact that learning objectives, assessment criteria and teacher support do not explicitly pay attention to multi-stakeholder collaboration and learning from multiple stakeholders. Students are often sent into the field without being properly prepared for the range of stakeholders involved, the perspectives these stakeholders represent, and the way students can mobilize and actively involve stakeholders in their projects. As Akkerman (2011) previously stressed that boundary crossing does not happen easily and needs explicit support, future research should focus on examining the effects of active and explicit support of student–stakeholder collaboration.

A high coaching intensity had a large positive effect on student learning in both studies. We stress the promising value of intense coaching, at least by stimulating critical reflection and explicating learning processes and experiences, for boundary crossing learning processes to actually occur (Sletto, 2010; Akkerman & Bakker, 2011; Bakker & Akkerman, 2014). Even more, explicating learning amongst all parties involved in the RLE would contribute to the ultimate aim of the RLE to collaboratively create new knowledge and expertise that supports sustainable regional development and/or transition by learning from and with each other (Foorthuis et al., 2012).

Limitations of this study are at first related to the comparison of the seven studied RLEs. The RLEs were actual learning environments in educational practice. As a consequence, the RLEs differed with respect to more than the three independent variables. They differed, for example, for the level of education, position of the RLE in the curriculum, duration of student projects, size of the student groups, specific learning objectives and assessment criteria. Though the effects of gender and age were controlled for, the other differing variables were not taken into account, while they might have influenced which competencies developed, and the degree to which these competencies developed. Excluding effects of the other varying elements would require an experimental design, which would be at the expense of the authenticity of the RLE. This study at least moves away from being just exemplary using a quasi-experimental design, that we regard to be the highest achievable type of systematic investigation (Wu & Brooks, 2011). A second note of caution should be made for the validity of the t-tests regarding the small numbers of participating subjects in some of the RLEs. Since we found highly significant results, despite these small numbers, in 10 of 12 examined RLEs, we think we found a real indication for the RLEs’ overall effectiveness. Finally, we realize that the choice for a systematic, basically quantitative investigation disabled for in-depth insights into what really happened in each RLE. As said above, future in-depth case studies could expose more detailed knowledge on students learning as a result of the typical characteristics of the RLE. Having confirmed the effectiveness of the RLE on the basis of two successive studies, we recommend planning educators to consider the inclusion of authentic multi-stakeholder learning environments in their curricula. To boost the development of planning relevant competencies in these learning environments, we recommend to let students work in multi-disciplinary student groups, to facilitate and stimulate intense collaboration with external stakeholders, and to organize an accommodating coaching trajectory to explicate learning from and between different disciplines and stakeholders. This study allows for providing evidence-based recommendations to innovate planning education and thereby contributes to the call for a better pedagogical fundament for innovations in authentic learning environments as used in planning education (Angotti et al., 2011; Balassiano & West, 2012; Long, 2012a; Nemeth & Long, 2012). From an educational research perspective, this study and its predecessor add to the recently launched theoretical debate and call for empirical studies delivering insights into the functioning of boundary crossing learning in education (Akkerman & Eijck, 2013; Bakker & Akkerman, 2014; Bronkhorst & Akkerman, 2016).