Scope, nature and progress of impact in practice-oriented educational research: a conceptual and empirical substantiation

ABSTRACT

Practice-oriented educational research is renowned for its impact, both in educational practice and research. Yet, existing studies on the impact of practice-oriented educational research reflect a proliferation of ideas on what impact is, can or should be. This study aims to contribute to an in-depth understanding by establishing a theoretically informed and empirically substantiated conceptualisation of the impact of practice-oriented educational research. Based on current literature, a tentative conceptualisation in the dimensions scope, nature and progress, representing the who, what and when of change, is proposed. The tenability and completeness of this conceptualisation is subsequently investigated in a qualitative empirical study. Based on interviews, individual reflections and small-group discussions, the impact of 10 purposefully selected practice-oriented studies in secondary STEM-education in the Netherlands is compared to the tentative conceptualisation of impact. This results in an empirical substantiation and extension of the tentative conceptualisation. The presented conceptualisation of impact of practice-oriented educational research in terms of scope, covering educational practice and research within the context of a study or beyond, nature, including conceptual, instrumental and symbolic change, and progress, comprising sustainability, timeframe, and stability of change, can facilitate and focus discussions, considerations and analyses of the impact of practice-oriented educational research.

KEYWORDS:

• Impact
• practice-oriented educational research
• educational practice
• educational research
• teacher research

Introduction

Practice-oriented educational research is renowned for its supposed impact in educational practice. In line with characterisations of practice-oriented educational research current in the Netherlands (where this study was conducted) and as explicated by the Netherlands Initiative for Educational Research, this study characterises practice-oriented educational research as scientific research emanating from an issue in educational practice, being conducted in practice with the collaborative involvement of relevant stakeholders (i.e., researchers and teachers), and building on and aiming to contribute to both educational research and practice (NRO 2021). These characteristics can cover a broad variety of research. Research can emanate from an issue in one school or be an issue on national or international scales. Stakeholders can play different roles in practice-oriented educational research. Teachers can be involved as researchers, or as designers or users of interventions. Researchers can have leading or supervising roles in a study. The (aspired) contributions to educational research and practice can also differ across studies. In this study, all varieties of research that align with the characteristics provided above, such as design-based research, action research, lesson study or practice-oriented educational research conducted by teachers, are considered practice-oriented educational research in this study.

Practice-oriented educational research is increasingly popular, as evidenced by the high share of practice-oriented educational research articles in scientific journals of education (Ertl et al. 2015). However, there is limited empirical research on, and hence limited empirical evidence, to support the notion of impact of practice-oriented educational research in educational practice (cf. Levin 2013), let alone on educational research. Most studies concerning impact focus on connections between educational research and practice (e.g., Huberman 1994; Kennedy 1997; Levin 2013; Farley-Ripple et al. 2018) or how research can contribute to educational practice (e.g., Weiss 1979; Dagenais et al. 2012). Searching for literature on the impact of practice-oriented educational research, we predominantly found studies on practice-oriented educational research conducted by teacher-researchers. These studies consistently emphasise impact on the teacher-researchers themselves (e.g., Henson 2001; Campbell and Jacques 2004; Snoek and Moens 2011; Bakx et al. 2016; Hilton and Hilton 2017; Dunn, Hattie, and Bowles 2019). Impact on broader contexts in educational practice and research remains largely unknown. In addition, discussions on impact tend to become increasingly focussed on ‘what works’, turning impact into a narrow instrumental concept. However, as convincingly argued by Biesta (2007, 2010), a broader interpretation of the contribution of educational research to educational practice and research is desirable. Such a broader interpretation should synthesise the proliferation of ideas on what impact is, can or should be currently reflected in the literature.

The aim of this study is to contribute to such an in-depth understanding of the impact of practice-oriented educational research to facilitate discussions, considerations and analyses of the impact of practice-oriented educational research studies, both by researchers and practitioners. In the next section, synthesising available literature, a tentative definition and conceptualisation of the impact of practice-oriented educational research are proposed. Subsequently, inspired by the idea of a ‘proof-of-concept’, the tentative conceptualisation is empirically substantiated in a qualitative study on the impact of purposefully selected practice-oriented educational research studies in secondary STEM-education in the Netherlands to establish its tenability and completeness. We should note that this elaboration focusses on the impact of specific practice-oriented educational research studies and not the impact of practice-oriented educational research in general (i.e., as a research approach, cf. Brown 2005). Furthermore, we acknowledge that the conceptualisation of impact presented in this article is particularly fitting with practice-oriented educational research as characterised above, most notably concerning the characteristic twofold aim to contribute to both educational research and practice.

Tentative definition and conceptualisation of impact of practice-oriented educational research

Definition of impact

Impact, also referred to as research use or educational change, is variously described in the literature as ‘the varied and unpredictable ways that research changes the way people think, how they understand, explore and reflect on their life-worlds’ (Saunders 2011, 16), ‘when research, in any of its multiple forms, makes a difference to subsequent actions that people take or refrain from taking’ (Levin 2004, 2), and ‘an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia.’ (Higher Education Funding Council England 2011, 48). From these descriptions of impact, we conclude that ‘change’ is a central characteristic of impact. Additionally, it is widely acknowledged that most impact does not occur in a direct manner, but in an indirect one (e.g., Weiss 1979; Bates 2002; Burkhardt and Schoenfeld 2003; Gardner 2011; Dagenais et al. 2012; Cain and Allan 2017) and can therefore take time to occur. Furthermore, research by Vulliamy and Webb (1992) shows that two aspects can induce impact, namely the product and process of research. Reflecting these notions and taking into account the characteristic twofold aim of practice-oriented educational research, we tentatively define impact of practice-oriented educational research as every change in educational practice or research that occurs at some point in time, whether or not intended, from the product or process of a practice-oriented educational research study.

Conceptualisation of impact

In a conceptualisation of the scale of educational reform, Coburn (2003) proposes four interrelated dimensions, namely the spread, depth, sustainability and ownership of reform. Inspired by these dimensions of reform by Coburn and taking into account discussions on the impact of practice-oriented educational research in the literature, impact is tentatively conceptualised in three dimensions: scope, nature and progress, representing the who, what and when of change.

Below, the three dimensions are theoretically elaborated based on conceptual and empirical studies on impact and use of (practice-oriented) educational research, whether or not conducted by teacher-researchers, and studies on educational change. A visual representation of the dimensions of impact, as discussed below, is presented in Figure 1.

Figure 1. Conceptualisation of impact of practice-oriented educational research.

Note: The division of ‘conceptual change’ in ‘cognitive’ and ‘affective’, and the subdimension ‘stability’ under the dimension ‘progress’ resulted from the empirical part of the study.

Scope

The first dimension of impact is scope, identifying the targets of change (i.e., the who). Following from the dual purpose of practice-oriented educational research, the scope of impact includes educational practice and educational research. This aligns with Zwart, Smit, and Admiraal (2015) who in their review study of teacher research include impact on school development and the scientific knowledge base.

Pertaining to scope, Frost and Durrant (2002) differentiate between impact within the school of a teacher-researcher and impact beyond the school. Impact within the school is focussed on individuals (i.e., teachers and students) and the school as an organisation. Impact beyond the school, on the contrary, is focussed on more abstract targets, such as public debate, transfer of professional knowledge and contributions to social capital. Two subdimensions of scope can be identified from this. First, impact can be in or beyond the context of research. Second, impact can be on individuals (i.e., teachers, students), a community (i.e., school) or society (i.e., public debate, social capital). Building on these ideas, we conceptualise the scope of impact as consisting of individuals and communities in the context of research or other contexts, both in educational practice and research, and on the fields of educational practice and research in general.

Illustrative of this conceptualisation of scope in educational practice is a study by Vrijnsen-de Corte et al. (2013) which shows that practice-oriented educational research conducted by teacher-researchers can contribute to impact on individual levels, such as improved student learning and achievement and increased professional development and motivation of teachers and ensuing changes in their teaching practice, and to impact on community levels, by changing the culture and enhancing educational development within a school. Additionally, Zwart, Smit, and Admiraal (2015) found that impact objectives for practice-oriented educational research conducted by teacher-researchers mainly focussed on changes in the context of study, such as the teacher-researchers’ professional development and changes in their educational practice and schools (Zwart, Smit, and Admiraal 2015). Impact in other practice contexts can also occur, for example when teacher-researchers present their research at conferences or write research reports to share their results with practitioners from other schools (Meijer et al. 2013).

Concerning impact on educational research, several studies indicate that practice-oriented educational research could contribute to educational research by adding new knowledge to the scientific knowledge base. Sometimes this succeeds (e.g., Admiraal et al. 2017), at other times it remains an intention (e.g., Meijer et al. 2013; Zwart, Smit, and Admiraal 2015). Besides impact on the field of educational research, studies by Huberman (1994, 1999) show that individual researchers do not remain unaffected by sustained interaction with practitioners during practice-oriented educational research studies, as it can lead to changes in their conceptions, research interests, methodological approaches and teaching. Similar to educational practice, the field of educational research is composed of individuals and communities. Impact on these individuals and communities could be valuable in itself and provide a plausible and interesting route towards impact on educational research in general. In line with the scope of impact in educational practice, the scope of impact in educational research therefore consists of individuals, communities and the field in general.

Nature

The second dimension of impact is nature, identifying the kind of change (i.e., the what). In a review on the use of research-based information by school practitioners, Dagenais et al. (2012) discuss three forms of change which they refer to as ‘research use’ by teachers. Conceptual change implies that research influences teachers’ thinking. Cain (2015) specifies that research can affect both the content and way of teachers’ thinking. Instrumental change implies that research influences teachers’ acting and decision making, leading for example to changes in teaching practice. Symbolic or strategic change, also termed confirmatory (Cain 2015) or political change (Tseng 2012), implies new or changed substantiations to maintain pre-existing opinions or practices. As for the latter, no actual changes in thinking or doing occur. Even though the above studies on the nature of change pertain specifically to teachers, we presume that these kinds of changes can also apply to other individuals or to communities in both educational practice and research. For example, Broadfoot and Nisbet (1981, 116) established that ‘the impact of research on educational studies is not limited to building up a substantive content to the discipline, but also and more importantly, research influences the style of thinking within these disciplines’, implying conceptual changes on individual and community levels in educational research. From this notion, we assert that practice-oriented educational research can just as well contribute to conceptual, instrumental and symbolic changes in individual researchers, communities of researchers and the field of educational research in general.

Progress

The third dimension of impact is progress, identifying the course of change over time (i.e., the when). Coburn (2003) argues that sustaining change over time is key to deep and lasting change. Hilton and Hilton (2017) found that teacher research can result in long-term change for the teacher-researchers involved, leading to lasting changes in educational practice in their classrooms and in their school, for example by increased collaboration with colleagues, continuing use of data or continuing research activities. Concerning impact beyond the context of research, Gardner (2011) argues that educational research in general mostly does not have an immediate impact but may take many years because ‘it needs to be interpreted and mediated in a variety of processes to accommodate different circumstances’ (p. 559), both in educational practice and research. We presume that this also applies to practice-oriented educational research. From these notions, two subdimensions of progress are identified: sustainability, referring to how long impact lasts, and timeframe, referring to when impact occurs.

Research question

To empirically substantiate the proposed conceptualisation of impact of practice-oriented educational research, a qualitative study into the impact of purposefully selected practice-oriented educational research studies conducted by postdoctoral teacher-researchers in the Netherlands, is conducted. The intent of the empirical study is to explore the tenability and completeness of the proposed conceptualisation. Since impact on teacher-researchers themselves has been studied extensively, this study focusses on impact in educational practice and research beyond the teacher-researchers. The central research question is:

What impact do the teacher-researchers describe for their practice-oriented educational research studies and how does that compare to the tentative conceptualisation of impact in the dimensions scope, nature and progress?

Method

In a qualitative study, empirical data on the impact of 10 teacher-researchers’ practice-oriented educational research studies were analysed deductively and inductively (Miles, Huberman, and Saldaña 2014) simultaneously, meaning that the dimensions and subdimensions of impact as described above were used as predefined codes while maintaining an open view to identify any other (sub)dimensions of impact that could emerge from the data.

Context

In the Netherlands, practice-oriented educational research is the most commonly used research approach, evident from funding patterns (NRO 2018) and researchers’ self-reported research activities (van Braak and Vanderlinde 2012). There is also a growing amount of practice-oriented educational research being conducted by teacher-researchers in doctoral and recently also postdoctoral research projects (Van Bergen, Groot, and van der Wel 2018). These research projects are largely subsidised by the Dutch Ministry of Education and the Dutch Research Council, because of the expected impact on educational practice, for example on professional development of teachers or quality of education.

As the empirical context for this study, we purposefully selected (Patton 2002) 10 postdoctoral research projects conducted by teacher-researchers. This context was selected based on the alignment of the teacher-researchers’ studies with the characteristics of practice-oriented oriented educational research, and the dual positions of the teacher-researchers within both educational practice and academia. The teacher-researchers’ projects were all design-based studies (c.f. McKenney and Reeves 2012; Bakker 2018) that emanated from an issue in educational practice, were conducted in educational practice with involvement of relevant stakeholders in educational practice (i.e., school-based colleagues) and academia (i.e., university-based supervisor), and building on and, paramount for this study, aiming to contribute to both educational practice and research. Furthermore, following from their dual positions, the teacher-researchers were assumed to be in a favourable position to reflect on the impact of their research projects. Their familiarity with the needs and procedures of both educational practice and research was assumed to contribute to their ability to articulate the aspired and achieved impact in both worlds.

The postdoctoral research projects were funded by the Ministry of Education. A programme committee consisting of three professors in the field of (science) education awarded research grants to teacher-researchers based on the quality of their research proposals, including their alignment with the characteristics of practice-oriented educational research (as outlined above) and consequent concern and potential for impact. The programme committee assigned the third author to organise bimonthly meetings for the postdoctoral teacher-researchers for purposes of peer support. At these meetings, experiences concerning their distinctive dual positions as teachers and researchers were shared, and the content and advancement of their research studies was discussed.

Respondents

Ten postdoctoral teacher-researchers were selected as respondents. The respondents were teacher-researchers with five to twenty-one years of experience as science or mathematics teachers in secondary education and a doctoral degree in either science, mathematics, or science or mathematics education. The teacher-researchers received a grant to conduct a practice-oriented educational research study for two days a week for two or three years while maintaining their teaching positions in secondary education. Table 1 provides an overview of the teacher-researchers and the titles of their research projects. The teacher-researchers conducted their research in their own schools with the support of a university-based supervisor. They all had a secondment at a university for the duration of their research project.

Table 1. Overview of teacher-researchers and their research projects.

Teacher-researcher	School subject	Title of postdoc research project
Anna	Chemistry	Interrelatedness of context-based chemistry education and student needs
Alex	Biology	A practical approach to within classroom differentiation using video’s
Daniel	Mathematics	Differentiation according to students’ interest in mathematics education
John	Biology	Influence of knowledge on neurological processes on teachers’ classroom practice
Leonard	Physics	Multidisciplinary contexts for flexible use of concepts
Mark	Biology	Professional development in a teacher development team focused on design, use and evaluation of context-concept education
Mike	Biology	Use of data to enhance teachers’ teaching practices
Oscar	Physics	Modelling in physics education
Peter	Physics	Technical internships to enhance students’ motivation for science
Tessa	Chemistry	Enhancing language proficiency of grade 10 students in science education

Data collection

Data were collected using two individual interviews, individual reflections, and small group discussions, leading to method triangulation (Miles and Huberman 1994). Multiple methods of data collection were used to provide a variety of opportunities for the respondents to consider the impact of their research. Since the teacher-researchers’ studies were ongoing at the time of this study, it was expected that the teacher-researchers had clear ideas about the aspired impact of their studies and that this aspired impact might have been partially achieved already. Both aspired and achieved impact were therefore included.

At the start of the teacher-researchers’ research projects, the first individual semi-structured interview was conducted. The teacher-researchers were asked about the goals of their research project, for example: What do you want to achieve with your research project? For yourself? For the school? For the university? When would you be satisfied with your research project? The interviews were recorded and transcribed.

In the middle of the second year, the teacher-researchers were asked to write an individual reflection on the impact of their research project. They received an empty matrix in which they could write down their aspired and achieved impact in educational practice and research. These reflections were held at one of the teacher-researchers’ bimonthly meetings. After writing down their individual reflections, the teacher-researchers engaged in small group discussions with two or three teacher-researchers to discuss differences and similarities in their aspired and achieved impact. The individual reflections were collected, and the small group discussions were recorded and summarised.

At the end of the second year, a second individual semi-structured interview was held with each teacher-researcher. They were again asked about the aspired and achieved impact of their research, for example: What is most important for you to achieve with your research? Are there any other things you would like to achieve? Are there any unplanned or unforeseen achievements of your research? The teacher-researchers were also asked to elaborate on the statements in their individual reflections, for example: What do you mean by this impact in educational practice? Why do you want to achieve this impact on educational research? To ensure overall coverage of the dimensions scope, nature and progress of impact and their subdimensions, an overview of the (sub)dimensions of impact was presented to the teacher-researcher at the end of the interview to initiate discussion of (sub)dimensions of impact not addressed so far. The interviews were recorded and transcribed.

The respondents were informed about the purpose of the data collection and the aim of this study upon invitation to participate and at the start of the first interview. All respondents verbally consented to participate. During the subsequent moments of data collection, the purpose of the data collection and aim of the study were restated and all participants continually consented to participate. This was in line with rules and regulations for this type of research in the Netherlands at the time the study was conducted.

Data analysis

Data were analysed using a qualitative content analysis strategy (Schreier 2013). First, a coding scheme was developed with the dimensions scope, nature and progress of impact and their subdimensions as described in the theoretical elaboration as categories and subcategories. Subsequently, descriptions of all categories and subcategories, examples and decision rules were added to the coding scheme.

Second, all quotations containing information on aspired and achieved impact and pertaining to any of the (sub)categories in the coding scheme were selected, summarised in a descriptive statement and coded. An example of this is presented in Table 2. During this process of deductive coding, we maintained an open view to identify any new (sub)dimensions of impact that could emerge from the data. The predefined coding scheme was adapted following the identification of new subdimensions of nature and progress of impact.

Table 2. Example of quotation, summarising descriptive statement and coding.

Quotation	Summarising descriptive statement	Coding
Q: What do you hope your research will yield for your school? A: Well, the wish to teach in a more differentiated manner is substantive, for everyone, but almost nobody actually succeeds at it. It would be really nice if we could achieve something, like say 10% of all lessons, or 20%, are taught in that way. Look, you should never do something for 100%. But that everyone has some guiding principles to teach in a differentiated manner. (Alex, interview I)	Alex wants all teachers at his school to get guiding principles to teach in a differentiated manner, so that they can change 10% to 20% of their lessons to more differentiated education.	SCOPE = practice Context = of research study Target = individuals teachers NATURE =  Conceptual, cognitive Instrumental PROGRESS = /

The third step was data condensation. All quotations with the same codes were merged and summarised in descriptive statements of impact. An overview matrix (Miles, Huberman, and Saldaña 2014) containing the descriptive statements pertaining to each dimension of impact was created for each teacher-researcher. In a last step, all overview matrices were checked against the adapted coding scheme to ensure all data fitted the coding scheme. All data fitted the coding scheme and the overview of dimensions and subdimensions of impact was therefore considered complete.

Quality of the data analysis was checked using an audit procedure as described by Akkerman et al. (2008) and extended by de Kleijn and van Leeuwen (2018). A peer researcher not involved in the study performed a summative audit. The auditor considered the data analysis procedure to be visible, comprehensible and acceptable, and the ensuing description of results to correspond with the data.

Results

The results are presented as an empirical substantiation of the proposed conceptualisation of impact of practice-oriented educational research in the dimensions scope, nature and progress. In the following section, all discussions of impact could concern aspired or achieved impact as both kinds of impact were included in our data collection and analysis. As it is irrelevant for the aim of this study whether impact was aspired or actually achieved, and for purposes of readability, we refrain from explicating this throughout the text.

Scope

All teacher-researchers discussed impact on both educational practice and research. They referred to impact on individuals (e.g., students, teachers, researchers), communities (e.g., physics team, school, research group) and on more abstract targets in educational practice and research in general (e.g., policy, knowledge base). The individuals and communities as targets of change could be located in the context of study or beyond. The teacher-researchers’ descriptions of the impact of their practice-oriented educational research studies thus aligned with the proposed subdimensions of the scope of impact.

The teacher-researchers typically discussed impact on individuals and communities in their own school and on the field of educational research by adding to the scientific knowledge base most elaborately. However, the teacher-researchers also discussed impact in educational practice beyond their own school and in general, and on individuals and communities in educational research. Exemplary quotations to illustrate impact for different targets of change are presented in Table 3.

Table 3. Overview of teacher-researchers’ exemplary quotations to illustrate scope and nature of impact of practice-oriented educational research.

EDUCATIONAL PRACTICE				EDUCATIONAL RESEARCH
WITHIN CONTEXT OF STUDY OR BEYOND
INDIVIDUALS
conceptual change		instrumental change		conceptual change		instrumental change
Q1	Cognitive and affective change: ‘Well, multiple things. So, they [students] should be able to see that there are real-world problems for which you need different techniques and subject knowledge to solve them. Often students find those real problems interesting. So that is one thing, their motivation. But also, that they become more able to flexibly use their monodisciplinary subject knowledge. So, suppose that they learn something in chemistry and then suddenly they have to apply that in a very different way in biology, again, in a different context. If they can do that, I think their understanding becomes more flexible.’ (Leonard, interview 2)	Q3	‘I set the go al for myself to get as many students as possible to go to a technical university. That is my goal. So with this research I want to figure out if we are on the right path to achieve that. So not just following those educational innovations that are proposed on a national level, but doing what is right to achieve that goal.’ (Anna, group discussion)	Q5	Cognitive change: ‘What I think is valuable, is that I am working at school and at the university. I think that some university researchers should stand in front of a classroom some time. That would be a whole experience for them, because I really think they miss that experience. I think I can communicate that practice experience to them. I think that can be really valuable for people who work at the university.’ (Anna, interview 1)	Q7	‘The most important issue is that the research question is answered so that they [supervisors] can continue research on this topic.’ (Alex, interview 1)
Q2	Affective change: ‘Well, they [colleague teachers] are more interested in the approach in my teaching materials and the impact of my teaching materials than in my research. They are less interested in that, but more in things like differentiation according to interest and letting students make choices and the effects of that et cetera.’ (Daniel, interview 2)	Q4	‘The most important thing that I want to achieve is to develop a procedure that teachers can use for differentiated instruction over a longer period of time while maintaining grip and control.’ (Alex, interview 2)	Q6	Cognitive change: ‘Well, he [university-based supervisor] does not know so much about language, so for him there is also [something to learn]: the relationship between language and natural sciences.’ (Tessa, interview 1)
COMMUNITIES
conceptual change		instrumental change		conceptual change		instrumental change
Q8	Cognitive and affective change: ‘There now is someone who has the time to walk around in the school. I can take a look into their classrooms more often. I learned a lot from that, but they did as well, from observing. […] I observed other [physics] teachers, but they could also learn from that because we had discussions about it. How do you do something? Why do you do it that way? What are pros and cons? […] You should ask them, but I am certain that we all gained from it. And what stands out is that we now have a team in which everybody finds that the collaboration and atmosphere are great. I don’t think that is because of my person, but I think that is because of the fact one of us has the freedom to go to the others, to observe, to collaborate.’ (Oscar, interview 1)	Q10	‘What I would like is that higher order thinking. […] Teachers should support that by asking questions in a structured way, because students can’t do that on their own. […] Higher order thinking needs to be well implemented for students […] and that needs to be done by teachers. That is what I really want to achieve and what I want to become part of the regular curriculum.’ (Anna, group discussion)	Q13	Cognitive change: ‘When I am here [at the university] in this scientific community, and we have a research team that meets about once a month, I notice that I can share a lot from my experiences in practice. Because I notice that people here [at the university], they kind of have an idea about students and what happens in a school, but the atmosphere in a school and how hard it is to change something for example, well, they often don’t have a good image of that. Then I can bring in nuances or explanations. So, it is valuable to share here [at the university] what I encounter at school and what I encounter as a teacher in my classroom for example’ (Mark, interview 1)	Q15	‘One impact is that there is more attention for language in the teacher training program. I discussed this with my supervisor and he added language exercises to the classes for new teachers.’ (Tessa, interview 2)
Q9	Affective change: ‘I want to initiate a culture of learning in my school. That they understand that change does not occur in one leap, but that it happens in small steps. That requires a long-term vision from the school.’ (Mike, interview 1)	Q11	‘In the school’s mission statement, they state five principles. Well, if you want your education to connect to the development of the brain, which is one of the principles, you need knowledge about that. But knowledge, how to do that, is minimally present. So that is what I studied so I can say which are relevant points and what that means for the school’s policy and other school things, like what course materials or methods to purchase. So that informed decisions can be made.’ (John, interview 2)	Q14	Cognitive change: ‘At the university they really want to know how teacher development teams work, how they do and do not work. They want to use that knowledge to study other teacher development teams, both the success factors and the methodological aspects, like how to study the development of teachers.’ (Mark, interview 1)	Q16	‘I know that different people [in my research group] are doing different things with interdisciplinarity and I talk to them of course. And well, it could become more of a research theme, so that these [different things] can reinforce each other.’ (Leonard, interview 2)
		Q12	‘My intended impact is to improve the set-up of the internships, mainly by increasing the role of parents. Then we want the results […] to be tested in other schools and in other regions.’ (Peter, group discussion)
BEYOND CONTEXT OF STUDY
EDUCATIONAL PRACTICE IN GENERAL				EDUCATIONAL RESEARCH IN GENERAL
conceptual change		instrumental change		conceptual change		instrumental change
Q17	Cognitive change: ‘I think in general, schools, on a large scale, just do all kinds of things without being able to give substantive arguments for them. […] I hope I can contribute to [change] that.’ (John, interview 1)	Q18	‘It would be really nice, not that it is going to happen, if it would be expanded to the whole Netherlands so that all students in grade 8 or 9 students would do a business, social and technical internship.’ (Peter, interview 2)	Q19	Cognitive change: ‘My idea was to somehow scientifically prove that responding to students’ interest will improve their motivation and effort. So, if I can prove that, I’ve achieved something nice. And that by focussing their [students’] attention on how they can use mathematics in other disciplines, that you can improve their image of mathematics. And that you can integrate that in teaching materials. That is basically what I want to prove. […] If I look at the literature, I find some things on differentiation based on interest, but it is not the main focus. Differentiation is mostly based on ability levels, not students’ interest.’ (Daniel, interview 2)	Q20	‘What could it yield? […] That you should not just change so many things at the same time in giant leaps. […] Science should learn that you should not make drastic, big and many changes. Then at some point you have no idea anymore what you are changing and what effects it has. And before it [an educational innovation] is studied, it is already implemented. Maybe science should learn that it should be the other way around. That you should first know what you are doing before you implement it.’ (Anna, interview 1)

Concerning impact in educational practice, the teacher-researchers mostly left implicit whether they were referring to impact in their own school or in other educational contexts, as illustrated by Alex’ quotation (Table 3, Q4) on how he wanted ‘to develop a procedure that teachers can use.’ Teacher-researchers that did explicitly discuss impact in educational practice beyond their own school indicated similar impact in both contexts.

The teacher-researchers discussed impact in educational practice and research resulting from both process and product of their research. Concerning process, for example, Oscar (Table 3, Q8) stated that his doing research generated opportunities for mutual observation and collaboration in the physics team at his school, and Mark (Table 3, Q13) indicated that through his presence in the university research group, he was able to share experiences as a teacher. Concerning product of research, for example, Anna (Table 3, Q10) discussed how she wanted teachers to use the results of her study to support students’ development of higher order thinking skills, and Alex (Table 3, Q7) thought that his supervisors could use the results of his study to continue research on his topic.

As discussed, the teacher-researchers typically pursued impact in educational practice and research. The following quotation from Leonard illustrates that, from his perspective, these can be pursued simultaneously:

‘A framework [on interdisciplinarity] as a scientific outcome, but also practical for teachers, for education for students in which they can see that science subjects are interrelated, for a new way of working within the school. Those are really the most important things.’ (Leonard, interview 2)

Nature

The teacher-researchers addressed conceptual and instrumental change in educational practice and research on individual, community and field levels. The teacher-researchers did not address any symbolic change in educational practice or research beyond themselves.¹ The teacher-researchers’ descriptions of the impact of their research project did therefore not fully align with the proposed subdimensions of the nature of impact. Exemplary quotations of the teacher-researchers on conceptual and instrumental change at all levels in educational practice and research are presented in Table 3.

Conceptual change

Conceptual change refers to changes in the content and way of thinking on individual, community or field levels in educational practice or research. From the teacher-researchers’ discussions of impact, two distinctive natures of conceptual change can be discerned, elaborating the tentative conceptualisation based on the literature. Conceptual changes can be both cognitive and affective in nature. Concerning conceptual change in educational practice, the teacher-researchers discussed cognitive and affective changes for individuals and communities. On an individual level, cognitive changes could be increased knowledge or understanding for students, for example concerning language usage in science education, application of mathematics in various disciplines or career opportunities in the technical domain, or professional development for teachers, for example concerning use of videos to support student learning or use of data and inquiry to improve classroom practice. On a community level, cognitive changes could be the development of a more long-term vision on educational change within a school. Affective changes could be changes in students’ motivation or development of teachers’ interest in a certain topic, such as in differentiation according to students’ interest (Table 3, Q2). On a community level, affective changes could be an enhanced atmosphere within a teacher team (Table 3, Q8), increasing sense of community among teachers or a growing culture of learning within a school (Table 3, Q9). Leonard’s quotation (Table 3, Q1) illustrates how he simultaneously aspired to an affective change (i.e., motivation), and a cognitive change (i.e., flexible use of monodisciplinary subject knowledge), for his students.

Concerning conceptual change in educational research, the teacher-researchers only addressed cognitive conceptual changes. On an individual level, this could be changes in researchers’ knowledge on the teacher-researchers’ research topic, such as multidisciplinary science education or interrelatedness between students’ language proficiency and achievements in science education. There could also be changes in understanding of educational practice and how research might contribute to changes in educational practice, both on the level of individual researchers and on the community level of a research group.

Concerning conceptual change in the fields of educational practice and research in general, the teacher-researchers aspired to contribute to the practical and scientific knowledge bases. For example, John aspired to change the way schools substantiate educational reform (Table 3, Q17), and Daniel set out to add knowledge on differentiation according to students’ interest to existing knowledge on differentiation according to ability levels (Table 3, Q19).

Instrumental change

Instrumental change refers to changes in skills, actions, decisions or achievements. The teacher-researchers had clear ideas for instrumental changes at all levels in educational research and practice. On an individual level in educational practice, they aspired, for example, to improve students’ language skills, to increase their learning outcomes and achievements, or to change their study choice (Table 3, Q3). For individual teachers, the teacher-researchers aspired to change their classroom practice, such as use of differentiated instruction (Table 3, Q4) or use of appropriate feedback strategies towards students, and change in how they prepare for classroom practice, such as designing and evaluating new educational activities, discussing classroom practice with peer teachers, or collaborating with other teachers to improve educational practice. On a community level, the teacher-researchers discussed, for example, increases in collaboration within teacher teams, changes in curriculum (Table 3, Q10) and contributions to school policy and decision making (Table 3, Q11).

Concerning instrumental changes in educational research, the teacher-researchers addressed, for example, increased interaction between university-based researchers and educational practice, and that university-based researchers could use the results of the teacher-researchers’ research for future research (Table 3, Q7). On a community level, the teacher-researchers thought that their research could contribute to the research group, for example, by contributing to the set-up of a new research theme (Table 3, Q16) or by contributing to changes in a teacher education programme provided by the research group (Table 3, Q15).

Instrumental changes on the field levels of educational practice and research in general were focussed on changes in policy, such as the nationwide introduction of internships for students in secondary education in the Netherlands (Table 3, Q18) or reconsiderations of how educational changes are implemented (Table 3, Q20).

Progress

Progress was the dimension of impact least discussed by the teacher-researchers, but they did discuss a subdimension not explicitly discerned from the literature. In addition to sustainability and timeframe, they addressed the stability of impact, referring to the potential shift of impact over time. Exemplary quotations from the teacher-researchers on progress of impact are presented in Table 4.

Table 4. Overview of teacher-researchers’ exemplary quotations to illustrate progress of impact of practice-oriented educational research.

SUSTAINABILITY		TIMEFRAME		STABILITY
Q21	‘I expect it to be sustainable. […] They [teachers] have to name advantages and disadvantages of all steps of regular and differentiated teaching. The disadvantages of regular teaching are of course the motivators to do this [participate in intervention]. The advantages of differentiated teaching actually occur and the disadvantages are less than expected. [.] I already notice, and that is nice to see, that some colleagues become reluctant to teach in a regular way, old-fashioned lessons.’ (Alex, interview 2)	Q23	‘I really want it [published] in a good journal. For that you need some more content, you need to have something substantial. And that isn’t there yet. You really want data to underpin it. […] That is also why I haven’t yet presented my research in the research group. […] For my school it can all be a little less scientific. There you can share sooner, all small steps, compared to the research world. Everybody’s time is limited and then I really want to have something to say.’ (Tessa, interview 2)	Q27	‘Higher order thinking is what we actually want to achieve with this educational innovation, but teachers have to guide that process. […] A sort of scaffolding has to occur in their thinking. Students cannot do that on their own, that needs to be guided. I think that that is where the problem occurs for teachers. They think that students can do that on their own, but they need guidance. I think that teachers need to learn that first.’ (Anna, interview 2)
Q22	‘I think they [teachers] will remember that they participated in a nice course, you know? And I hope that the management will pay certain attention to use of the outcomes, to use the results in continuing development. […] I had a conversation about this with the school leader last Monday, about his vision for next year.’ (Mike, interview 2)	Q24	‘What I would like is that it results in a method for differentiated instruction and that it will be successful. […] In addition, I hope that the video’s become successful, because I have a strong believe in digitalisation and working with videos. […] So, in a little while, two to three years, we will have a whole new educational set-up [at school].’ (Alex, interview 1)	Q28	‘What I also noticed is that three teachers from my school [that participated in the teacher development team], start to collaborate more easily on other topics as well. […] They have a sort of common ground and I think that is nice to see. It gives them a feeling of, and sometimes I hear them talk about this, of belonging to a little club that knows something about practical inquiry and that has an inquiring attitude.’ (Mark, interview 2)
		Q25	‘I have not presented my research at a conference for chemistry teachers yet, because I do not have results yet.’ (Anna, group discussion)
		Q26	‘What I didn’t expect, and thus didn’t plan for, is, well, the expansion to other subjects. Of course, that is something you want, but you never know. That is always hard to predict, in what timeframe that will occur, but it is happening somewhat sooner than I expected.’ (Tessa, interview 2)

Sustainability

Ideas on the sustainability of the impact of their research varied among teacher-researchers. Quotations from Alex (Table 4, Q21) and Mike (Table 4, Q22) illustrate how the former believed that his impact is sustainable while the latter was less convinced of the sustainability and felt that action on behalf of the school management was needed to sustain impact beyond the duration of his research.

Timeframe

Teacher-researchers’ discussions of progress mainly related to impact in educational practice. Only Tessa addressed timeframe in relation to educational research. She discussed how impact in educational practice occurred sooner than impact in educational research (Table 4, Q23). The teacher-researchers differed in their views on when impact in educational practice could start, for example at the start of the research project or later when results are available (Table 4, Q25). Furthermore, ideas differed on when the full aspired impact can be realised. For example, Alex believed that it will take some years for the results of his research to become embedded in the whole school (Table 4, Q24), while impact in Tessa’s school was expanding sooner than expected (Table 4, Q26).

Stability

From the teacher-researchers’ statements a third subdimension of progress could be identified: stability. The teacher-researchers believed that impact can or should shift over time. For example, Anna’s primary aim was to increase students’ higher order thinking skills, but she believed that teachers’ scaffolding skills in this area needed to be addressed first. Subsequently, impact could shift to students to achieve her primary aim (Table 4, Q27). Impact could also shift beyond the intended impact of the teacher-researcher, as illustrated by Mark (Table 4, Q28). He noticed how teacher collaboration increased after participation in his intervention, even though this was beyond the aim of his intervention.

Conclusions, discussion, limitations and future research, and implications

Conclusions and discussion

This study aimed to contribute to an in-depth understanding of the impact of practice-oriented educational research. Following theoretical elaboration, the tenability and completeness of the proposed conceptualisation in the dimensions scope, nature and progress was investigated in an empirical study on the impact of exemplary practice-oriented educational research studies conducted by postdoctoral teacher-researchers. Overall, the results of this empirical study confirmed and complemented the tentative conceptualisation of impact and showed that the conceptualisation of impact in the dimensions scope, nature and progress is empirically tenable.

The impact of the exemplary practice-oriented educational research studies confirmed that the potential scope of impact of practice-oriented educational research covers individuals and communities in educational practice and educational research, in the context of study or beyond, and the fields of educational practice and research in general. However, not all subdimensions of the scope of impact were evenly discussed by the teacher-researchers. Concerning impact in educational practice, mainly impact in the context of research (i.e., in the context of the teacher-researchers’ own schools) was referred to. This is not surprising, considering the research studies originated from issues in the teacher-researchers’ educational practices. In general, impact beyond the immediate context of research remains unaddressed. However, similar issues can occur in other educational practices, making (the results of) the studies also relevant beyond the context of research.

Concerning the scope of impact in educational research, mainly impact on educational research in general by contributing to the scientific knowledge base was referred to. Again, this is not surprising, considering the high importance that is assigned to publishing in the domain of educational research. Moreover, contributing to the scientific knowledge base was often considered the only possible impact in educational research. However, our results indicate that impact in educational research can also pertain to individuals and communities within this field, potentially leading to valuable changes, such as increased understanding of educational practice and how research can contribute to it among university-based researchers, implementation of research results in teacher education, or changed research themes in research groups. We want to emphasise that neither we nor the teacher-researchers intend impact on individuals and communities in educational research to compensate for impact on educational practice: it is additional impact that, although usually not considered, is interesting and potentially valuable.

Furthermore, in line with the proposed conceptualisation of impact, the exemplary practice-oriented educational research studies showed impact of a conceptual and instrumental nature. The subdimension conceptual change was complemented by adding a subdivision in cognitive and affective change, thereby expanding the proposed conceptualisation of the impact of practice-oriented educational research. Conceptual changes of an affective nature were only discussed for impact in educational practice. This could have resulted from the teacher-researchers’ extensive experience in educational practice and more limited experience in educational research, making them more sensitive and articulate concerning changes in practice. It could also have resulted from the different values in both worlds and the more general disposition of educational research to focus on cognition.

The proposed conceptualisation of impact also includes changes of symbolic nature, but this subdimension did not come up in the teacher-researchers’ descriptions of the impact of their practice-oriented educational research projects. Lack of occurrence of symbolic changes in the results could be an artefact of our study, in which we collected data on aspired and achieved impact. This was mainly focussed on actual (conceptual and instrumental) changes in educational practice and research, because that is what is usually aspired. Furthermore, achieved symbolic impact beyond themselves would be difficult for the teacher-researchers to identify because of the characteristic of symbolic changes that there are no actual changes but merely new or changed substantiations for pre-existing conceptions or activities.

Progress of impact was shown to relate to sustainability and timeframe, and stability was introduced as a third subdimension. The impact of the exemplary practice-oriented educational research studies showed that impact can last over time (or not), can occur at different moments during or after completion of a study, and that changes can shift over time. Shifts in impact can be intended or move impact beyond aspirations. Stability as a subdimension of progress of impact was explicitly identified in our empirical study. However, in retrospect, potential shifts in impact are also implicitly suggested in the literature. For example, Vrijnsen-de Corte et al. (2013) suggest that teacher research should emphasise impact on teachers’ professional development which, in time, can lead to improved student learning and results, and Berger, Boles, and Troen (2005) suggest that teacher research can affect school culture, leading to changes in teachers’ teaching and subsequent students’ learning. Although shifts in impact are thus discussed in existing studies on impact of practice-oriented educational research, they are rarely conceptualised as such (cf. Engeström 2011).

The progress of impact in educational research was only discussed to a very limited extent. This could result from the respondents’ general inclinations to address impact in educational research less elaborately than impact in educational practice. Since the teacher-researchers mainly discussed impact on the scientific knowledge base, it could also reflect the common assumption that sustainability, timeframe and stability of impact in this area typically follow a standard course over time, for example concerning publication in scientific journals.

Furthermore, there is a generally held belief that practice-oriented educational research should result in different products to achieve impact in both educational practice and research (e.g., McKenney and Reeves 2012; Akkerman, Bronkhorst, and Zitter 2013) due to different requirements and values in both worlds and due to differences in aspired impact for both. However, results from our empirical study indicate that, for the teacher-researchers, these might not be as far apart as is often assumed. We assert that it is possible to strive for impact in both worlds simultaneously if the intended impacts align to some extent.

In general, the scope and nature of impact were typically discussed by the teacher-researchers; as already mentioned, progress was discussed considerably less. This could be an artefact of our study in which data collection was mainly centred around questions on aspired and achieved impact, eliciting responses on scope and nature of impact. It could also result from a (perceived) lower urgency of this dimension of impact, because scope and nature (i.e., who and what should change) need to be established first before any statements can be made about progress. However, we contend that the three dimensions of impact are inextricably connected and equally important to address when discussing impact of practice-oriented educational research. Discussions about the scope of impact become meaningless without reference to the nature of impact and vice versa, and discussions of impact with a certain scope and nature loose significance without reference to its progress.

Limitations and future research

The empirical study complemented the conceptualisation of impact of practice-oriented educational research. (i.e., added changes of cognitive and affective nature, and stability of progress). However, it remains uncertain whether this completes the conceptualisation. The analysed exemplary practice-oriented educational research studies were all design-based research studies, a specific kind of practice-oriented educational research. More diverse empirical contexts involving other practice-oriented research approaches, such as action research or lesson study, could yield other elaborations of the dimensions of impact. This could lead to expanded ideas about scope, nature and progress of impact.

In addition, the respondents in our empirical study were postdoctoral teacher-researchers. Since we focussed on impact of practice-oriented educational research, we presumed them to be in a favourable position to reflect on the aspired and achieved impact of their research. In addition, we assumed that it was irrelevant for our study who conducts the practice-oriented research, teacher-researcher or university-based researcher. From our results, we derive that the teacher-researchers were suitable respondents since they provided substantiations for all dimensions of impact and even complemented them by adding new subdimensions. Moreover, they were able to give elaborate insight in the aspired and achieved impact of their research studies in educational practice due to their experience of working within a school, with students and colleague teachers. However, concerning impact on individuals and communities in educational research, they were less articulate. This difference could result from their primary and permanent positions and extensive experiences as teachers, and their secondary and temporary positions and more limited experiences as researchers. Other non-teacher researchers could elaborate interpretations of impact in educational practice and research following from their positions and experiences. Future research could also explore how the conceptualisation of impact resulting from this study compares to (aspired) impact of practice-oriented educational research conducted by non-teacher researchers.

Furthermore, the conceptualisation of impact discussed in this study pertains to practice-oriented educational research with particular characteristics. We characterise practice-oriented educational research as emanating from an issue in educational practice, being conducted in educational practice with involvement of relevant stakeholders (i.e., researchers and teachers) and building on and aiming to contribute to educational practice and research. Our conceptualisation of impact does not necessarily apply to practice-oriented educational research with other characteristics, e.g. with a more or less prominent focus on contributing to educational research and/or practice. Future research could identify how conceptualisations of impact differ for practice-oriented educational research with other characteristics.

Lastly, both aspired and achieved impact on educational practice and research were taken into account in our data-analysis. Considering the aim and research question of this study, we considered it irrelevant whether impact was actually achieved since ideas on aspired impact could also elaborate the dimensions of impact. A focus on achieved impact in future studies could further substantiate, and potentially elaborate, the proposed conceptualisation.

Implications

At the beginning of this article, we introduced a tentative definition and conceptualisation of the impact of practice-oriented educational research. Based on extant knowledge and an empirical study, an empirically grounded conceptualisation of impact is presented in Figure 1. Following this conceptualisation, the tentative definition of impact can be further specified as change in terms of scope, nature and progress in educational practice or research that occurs, whether or not intended, from the product or process of a practice-oriented educational research study.

The presented conceptualisation adds a broader interpretation of the impact of practice-oriented educational research to existing discussions in the literature, that typically address only one dimension of impact or qualify impact in terms of level or extent (Dagenais et al. 2012). It differs from existing conceptualisations in that it focusses on scope, nature and progress of change in both educational practice and research. It thereby creates the possibility to discuss the impact of a practice-oriented educational research study in a structured and concise way without making any normative or subjective assertions about the level or extent of impact upfront. Although discussions on impact typically focus on positive changes, the presented conceptualisation can also facilitate discussions on undesired impact, retrospective concerning achieved impact but also prospective concerning aspired impact, i.e. what impact is to be avoided. What impact is (un)desirable is inherently normative and requires professional judgment by relevant stakeholders in practice-oriented educational research studies.

Furthermore, by separating the who, what and when of change from how to achieve it, the presented conceptualisation of impact can support practice-oriented educational researchers in explicating the aspired impact of their research without being prescriptive in how to achieve it and consequently imposing impediments on aspired impact. Practice-oriented educational researchers, both teacher-researchers and university-based researchers, can use the conceptualisation of impact presented in this study to ‘imagine the full range of impacts and to plan for them more deliberately’ (Frost and Durrant 2002, 151) over the course of their research studies. Similarly, practitioners can use the presented conceptualisation of impact to identify what impact can be expected or aspired to when implementing research findings. The conceptualisation offers an overview of possible directions for impact; it is not a guide that indicates what impact to achieve or prioritise. By supporting practice-oriented educational researchers and practitioners in establishing clear impact objectives, the presented conceptualisation of impact can contribute to achieving impact of practice-oriented educational research in both educational practice and research.

Acknowledgments

The authors thank Maarten Pieters for his thorough audit of our data analysis procedure.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the Ministerie van Onderwijs, Cultuur en Wetenschap under grant number 804AO-42742.

Notes on contributors

S.E.A. Groothuijsen

Suzanne Groothuijsen is a Postdoctoral Researcher at the Eindhoven School of Education at the Eindhoven University of Technology in the Netherlands. After finalising her PhD research on quality and impact of practice-oriented educational research, her current work is focussed on supporting evidence-based educational innovation with ICT in higher engineering education.

L.H. Bronkhorst

Larike H. Bronkhorst is an Assistant Professor at the department of Education of Utrecht University. Larike’s research explores learning and interest development across different contexts in and outside of education and the dynamics of research-practice collaborations from a boundary crossing perspective.

G.T. Prins

Gjalt Prins is an Assistant Professor in science education at the Freudenthal Institute for Science and Mathematics Education at Utrecht University in the Netherlands. His research focuses on education for sustainable development by means of design-based research. He is involved in several pre- and in-service teacher training programs, as well as governmental funded projects in which science teachers combine teaching tasks with conducting practice-oriented educational research.

W. Kuiper

Wilmad Kuiper is Emeritus Professor of Curriculum Evaluation at the Freudenthal Institute for Science and Mathematics Education at Utrecht University in the Netherlands. From 2008 till 2018 he was the head of the Curriculum Research Department at the Netherlands Institute for Curriculum Development (SLO).

Notes

1. One of the teacher-researchers describes symbolic change in his own classroom practice. He uses his research project to justify use of a pre-existing educational activity. Hence, there is no actual change, only new arguments to justify an existing practice. Since this concerns symbolic change of (the practice of) the teacher-researcher himself, this is not included in the results of this study.