Teacher-researchers’ quality concerns for practice-oriented educational research

ABSTRACT

Practice-oriented educational research is increasingly gaining traction in educational research due to its intention to contribute to both educational research and educational practice. Educational researchers have established quality concerns that practice-oriented educational research should meet in order to realise this intention. We argue that teachers’ quality concerns probably differ from researchers’ concerns. This may explain why practice-oriented educational research faces challenges concerning its contribution to educational practice. The aim of this study is to identify teacher-researchers’ perspectives on the quality of practice-oriented educational research and to analyse how these differ from the research perspective. In a qualitative empirical study, individual reflections, small-group discussions and semi-structured interviews of ten purposefully selected teacher-researchers are analysed following a so-called informed grounded theory approach. The results of this study show that the teacher-researchers’ quality concerns overlap with the quality concerns commonly held by researchers, but they broaden the meaning of some quality concerns, add new concerns and exclude others. Taking their common quality concerns as a starting point, close collaboration between researchers and teachers could decrease researchers’ challenges concerning legitimacy and relevance of their work and increase teachers’ use of research in educational practice.

KEYWORDS:

• Quality concerns
• practice-oriented educational research
• teacher-researchers
• research perspective
• practice perspective

Introduction

Practice-oriented educational research is increasingly gaining traction in educational research due to its explicit intention to build on and contribute to both educational research and educational practice. Oancea and Furlong (2007, 124) define practice-oriented educational research as ‘situated between academia-led theoretical inquiry and research-informed practice, and consisting of a multitude of models of research explicitly conducted in, with, and/or for practice’. Examples of practice-oriented research approaches are design-based research (e.g., van den Akker et al. 2006), action research (e.g., Townsend 2013), and lesson study (e.g., Watanabe 2002).

Since practice-oriented educational research is intended to contribute to educational research and practice, it is argued by several researchers that quality criteria for this kind of research should reflect this twofold objective. Oancea and Furlong (2007), Ros and Vermeulen (2010), and Verschuren (2009) discuss that the quality of practice-oriented educational research should be assessed based on scientific and practical standards. On a more general level, Gibbons et al. (1994) state that for application driven research ‘traditional scientific criteria will have to be qualified by other criteria that claim equal legitimacy’ (Gibbons et al., p. 153). From these arguments and the consensus that practice-oriented educational research is intended to address teachers in addition to researchers, we contend that the quality of practice-oriented educational research should be addressed from the perspectives of both researchers and teachers.

Discussions in the current literature of the quality of practice-oriented educational research are predominantly informed by researchers’ perspectives. The value of teachers’ perspectives is widely recognized (e.g., Cochran-Smith and Lytle 2009; Gore and Gitlin 2004; Ratcliffe et al. 2005; Winch, Oancea, and Orchard 2015). Although there are empirical studies of teachers’ perspectives on educational research, we could not find empirical research studies informing us about teachers’ perspectives on the quality of educational research in general or practice-oriented educational research in particular. Researchers and teachers typically operate in two distinct worlds and pursue fundamentally different activities (Hammersley and Gomm 2002). Notably, researchers study educational practice from an outsider perspective in contrast to teachers, who operate in educational practice and view it from an insider perspective (Kemmis 2012). Owing to their differing activities and frames of reference, researchers and teachers have different perspectives on educational research. They hold different epistemological beliefs (Joram 2007) and have different ways of using and validating research (Bartels 2003; Ratcliffe et al. 2005). From this it can be inferred that researchers and teachers have differing perspectives on the quality of practice-oriented educational research. Insight into teachers’ perspectives enables practice-oriented educational researchers to take these into account in their research and research reports (Hammersley and Gomm 2002). This can contribute to making practice-oriented educational research more relevant to both research and practice (Gore and Gitlin 2004; Kemmis 2012).

This study identifies the perspectives of purposefully selected teacher-researchers on the quality of practice-oriented educational research, and analyses how these differ from the research perspective portrayed in the current literature. In the next section, the concept of quality, research perspectives on the quality of practice-oriented research and teachers’ perspectives on educational research are discussed. Subsequently, a practice perspective on the quality of practice-oriented educational research is identified in a qualitative empirical study in which teacher-researchers were the informants.

Theoretical framework

The concept of quality

There is no unified definition of quality, in research or in general. Despite this lack of a definition, Wittek and Kvernbekk (2011) make three observations about the concept. First, quality is considered a property as it is something that can be attributed to something else. Second, quality implies a value judgment and is often expressed in quantitative terms e.g., quality can be high or low, increase or decrease. Third, quality can be viewed as subjective i.e., in the eye of the beholder, or as objective i.e., an entity that can be separated into specific measurable parts.

In discussions of the quality of educational research in the current literature, quality is mostly portrayed as objective and defined in terms of criteria that are considered to be explicit, comprehensive and observable (Hammersley 2007) for instance, internal validity, applicability and cumulativity. However, Hammersley disputes whether the quality of educational research can simply be portrayed in terms of explicit and concrete quality criteria. He presumes that there is always judgment involved based on a personal frame of reference, implying a subjective view of the quality of educational research. Oancea and Furlong (2007) also dismiss the idea of universal quality criteria or standards for educational research, because such universal criteria or standards do not recognize diversity of perspectives and do not do justice to the great diversity of educational research.

Furthermore, just like quality itself, quality criteria have no fixed meanings either. For example, a well-known criterion like internal validity is diversely described as ‘correspondence with empirical reality’ (Verschuren 2009, 15), ‘whether results found by the researcher are reliable and accurate for the phenomena that are being studied’ (Ros and Vermeulen 2010, 4) and ‘isomorphism or verisimilitude between the data of an inquiry and the phenomena those data represent’ (Guba 1981, 80). Even when criteria are described in discrete terms and arranged in separate parts, there will never be absolute clarity about when they apply and when they do not apply (Wittek and Kvernbekk 2011). For example, the quality criterion ‘cumulativity’ is generally meant to indicate that a study builds on previous knowledge and adds something new to the scientific knowledge base. However, how much previous knowledge should be integrated in a study? Few would consider it sufficient to build on one previous study. What if no recent studies are included or if only recent studies are included? What if only non-peer-reviewed references are included? Owing to this lack of clarity, discussions of the quality of educational research will always involve some form of subjective judgment rooted in personal frames of reference (Hammersley 2007).

From this line of reasoning, it becomes evident that only stipulative definitions of quality are feasible. Stipulative definitions are fitting for a particular context and discourse, within a certain frame of reference (Wittek and Kvernbekk 2011). This makes definitive shared definitions of the quality of practice-oriented educational research from either perspective troublesome. The perspectives on quality discussed in this study are therefore stipulative definitions. They are snapshots of current literature and the particular context of the empirical study and should be considered as tentative. We therefore claim modesty for the scope of the results of our study.

Research perspectives on the quality of practice-oriented educational research

In the literature, different overviews of quality criteria for practice-oriented educational research are presented. These lists of quality criteria differ in content, length and specificity. Generally speaking, two kinds can be discerned: the first focus on the quality of a research study in itself and (potential) effects on educational practice; the second focus on the procedures of research¹. For the purpose of our study, we selected three exemplary publications of the first kind. First, Oancea and Furlong (2007, 125) describe three so-called ‘domains of quality’ for practice-oriented educational research; an epistemic domain on methodological and theoretical soundness of research; a technical and economic domain on the (potential) value for the use of research; and a practical domain on the practical wisdom and (potential) enhancement of action by practitioners. They note that the specific quality criteria can vary for each practice-oriented research study, but that aspects from all domains of quality should always be considered. Second, Verschuren (2009) discusses methodology for practice-oriented research in the social sciences. This contribution is considered relevant owing to its elaborate discussion of quality criteria. He distinguishes between theory-oriented and practice-oriented quality criteria, or as he states it ‘between epistemological and implementary validity, or between truth and utility’ (p. 24). Third, Ros and Vermeulen (2010) establish quality criteria for practice-oriented educational research based on Oancea and Furlong (2007) and Verschuren (2009). They distinguish between scientific research standards and usefulness standards.

Table 1 provides an overview of the quality concerns for practice-oriented educational research in the three exemplary publications and the resulting research perspective on the quality of practice-oriented educational research. The starting point for the overview is the ‘four major concerns relating to trustworthiness’ of educational research by Guba (1981, 79). According to Guba, his four concerns about trustworthiness are independent of research paradigms and methods. Even though he translates the concerns into different quality criteria for rationalistic and naturalistic research and recommends different strategies and methods for quality control in both paradigms, the quality concerns remain the same. Guba’s quality concerns are: truth value i.e., the extent to which the results of a study are a correct representation of (experienced) reality; applicability i.e., the extent to which the results of a study have applicability in other contexts and/or with other subjects; consistency i.e., the extent to which a study and its results can be consistently repeated if the research were replicated; and neutrality i.e., the extent to which the results of a study are solely a function of the subjects and conditions of the research.

Table 1. Overview of the quality concerns in the literature and the resulting quality concerns from the research perspective

Guba (1981)	Oancea & Furlong (2007)	Verschuren (2009)	Ros & Vermeulen (2010)	Quality concerns from the research perspective
	Epistemic domain	Internal scientific (theory-oriented) criteria	Scientific research standards	Research-focused quality concerns
Truth value Applicability Consistency	Trustworthiness Propriety Transparency & explicitness	Internal validity External validity Verifiability	Internal validity External validity Accountability	Truth value	Extent to which the results of a study are accurate and reliable, and correspond with empirical reality
				Applicability	Extent to which the results of a study are generalisable to other contexts and/or other subjects
				Consistency	Extent to which a study’s research design and methodological procedures are transparent and explicit so that it can be replicated
Neutrality				Neutrality	Extent to which findings are a function solely of subjects and conditions of research
	Contribution to knowledge	Cumulativity	Cumulativity	Cumulativity	Extent to which a study builds on and contributes to scientific knowledge
	Practical domain	External scientific (practice-oriented) criteria*	Usefulness standards	Practice-focused quality concerns
	Plausibility Engagement Reflexivity, deliberation & criticism Receptiveness	Needs of stakeholders	Comprehensibility Acceptance & legitimacy	Comprehensibility	Extent to which practitioners can understand the research (results)
		Comprehensibility Acceptability Legitimacy
				Acceptability	Extent to which practitioners find the research (results) legitimate
	Transformation & personal growth	Research as learning process	Learning opportunities	Value for learning	Extent to which research (results) offer opportunities for professional development and capacity building
	Technical domain	Type of knowledge criteria		Usability	Extent to which research (results) provide solutions to practical problems
	Fitness to purpose Salience/timeliness Specificity & accessibility Concern for impact Flexibility & operationalizability	Holism Interdisciplinarity Context restraint Profoundness Manipulability
	Economic domain			Economic value	Extent to which research (results) contribute to economic advantages or (inter)national competitiveness
	Marketability & competitiveness Cost Auditability Feasibility Added value/brand

^aVerschuren’s criteria for ‘object of research’ are excluded from this synthesis because they ‘are not criteria, but features of the object that ask for methodological attention’ (2009, 17).

The four quality concerns expressed by Guba (1981) fit in the domain of research-focused criteria as distinguished by Verschuren (2009) and Ros and Vermeulen (2010), and the epistemic domain of Oancea and Furlong (2007). The criteria in these domains show considerable overlap with Guba’s quality concerns ‘truth value’, ‘applicability’ and ‘consistency’. The criterion ‘neutrality’ is addressed only by Guba; the criterion ‘cumulativity’ is addressed only by the other authors. Both criteria are included as they differ substantially from the other three criteria.

The remaining quality criteria fit within the domain of practice-focused quality criteria. The criteria ‘comprehensibility’, ‘acceptability’ and ‘legitimacy’ are present in the lists of quality criteria by both Verschuren (2009) and Ros and Vermeulen (2010), and they overlap with the practical domain of Oancea and Furlong (2007). They are therefore included in the research perspective. Following Ros and Vermeulen, ‘acceptability’ and ‘legitimacy’ are combined into one criterion that we term ‘acceptability’. The practical domain of Oancea and Furlong (2007) also refers to the transformation and growth of practitioners. This aligns with the learning opportunities as described by Ros and Vermeulen (2010) and the criterion ‘research as a learning process’ as described by Verschuren (2009). This marks a third criterion of ‘value for learning’. A fourth criterion is derived from the technical domain of Oancea and Furlong (2007) and the criteria for ´type of knowledge´ by Verschuren (2009). These refer to the potential for and conditions of use to solve practical issues and are combined in the criterion of ‘usability’. The last criterion is ‘economic value’ as derived from the economic domain of Oancea and Furlong (2007) about research’s ‘value for money’.

Generally speaking, all authors seem to hold objective views of quality, transpiring from the lists of criteria that they portray. However, these lists can be considered subjective because different authors include different criteria and attach different meanings to similar criteria. Owing to this subjectivity and the objectivity that the term ‘criteria’ seems to imply, the term ‘quality concerns’, as established by Guba (1981), seems more appropriate and is used in this study from this point onwards. The overview of quality concerns as displayed in Table 1 covers the common ground in the current literature and can be considered a stipulative definition of the quality of practice-oriented educational research from the research perspective.

The concerns in Table 1 are purposefully briefly described. Fitting with a subjective view of quality, it is assumed that every beholder holds a personal view of the meaning of these quality concerns. Providing exhaustive descriptions of these concerns that fit with everyone’s personal view would be impossible, because, depending on paradigm and epistemological beliefs, views can contradict. It would also be inappropriate to impose fixed meanings of these quality concerns, because every beholder is entitled to a personal view and no view is more legitimate than any other in this respect (Wardekker 2000).

Teachers’ perspectives on educational research

Teachers’ perspectives on the quality of educational research in general or practice-oriented research in particular could not be found in the current literature. However, teachers’ perceptions of educational research in general are addressed and provide insights into teachers’ concerns. Studies of this issue show a fairly uniform picture of what the aim and function of research should be according to teachers. First, research findings should resonate with teachers’ professional experience in educational practice (Everton, Galton, and Pell 2000; Labaree 2003; Ratcliffe et al. 2005) so that they can integrate research-based knowledge into their personal knowledge base without too much difficulty (Bartels 2003). If research findings conflict with teachers’ experience, teachers are inclined to oppose these findings and to give precedence to their own experience. They feel free to dismiss research-based information because, in their views, research does not have greater authority than their own experiences (Cain 2017). Furthermore, teachers tend to have a case-based way of thinking about educational practice, which means that they consider each teaching situation to be unique (Joram 2007). This leads teachers to dismiss general research-based knowledge and to assign privilege to personal professional experience (Labaree 2003).

Second, research should apply to teachers’ contexts (Gore and Gitlin 2004; Vanderlinde and van Braak 2010), which means that problems addressed in research should align with problems experienced in practice, and results should offer solutions to problems experienced in teachers’ specific contexts.

Third, research should make clear how findings or research-based interventions are to be used in educational practice (Gore and Gitlin 2004; Ratcliffe et al. 2005). Everton, Galton, and Pell (2000) found in their study of teachers’ perspectives on educational research that, according to teachers, research should focus on classroom action, specific aspects of teaching and demonstrate effective learning. Teachers need specific information about the components of an intervention, student engagement and measurements of success (Carnine 1995) to be able to use the results of research in practice. This aligns with studies of teacher decision-making in educational innovation that show that proposals for educational change should clearly describe procedures for classroom practice, be congruent with teachers’ perceptions of educational practice, and have benefits that outweigh the costs of changing educational practice (Doyle and Ponder 1977).

Last, research findings should be accessible for teachers (Carnine 1995; Gore and Gitlin 2004). This means that teachers should be able to access research reports physically or digitally, and that the level of difficulty and complexity of research reports should align with teachers’ levels of understanding. According to Bartels (2003) and Vanderlinde and van Braak (2010), teachers experience difficulty with language and jargon in research articles.

Unsurprisingly, teachers’ primary focus is their own educational practice. In contrast to researchers, who establish quality criteria for educational research on a community level, teachers establish criteria for quality on an individual level (Bartels 2003) based on their own professional knowledge and experience. It also shows that teachers are looking for a different kind of knowledge from what (most) research has to offer (McIntyre 2005). Teachers are looking for pedagogical (content) knowledge instead of propositional knowledge, and they assign priority to the practicality of research in contrast to researchers who are looking for clarity, coherence and truth. From these differences it can be discerned that teachers have different concerns about the quality of practice-oriented educational research from researchers. Their concerns about educational research in general, as described above, provide some preliminary directives for their concerns about the quality that practice-oriented educational research should address to find resonance with teachers.

Research aim

The aim of this study is to identify teachers’ perspectives on the quality of practice-oriented educational research and to analyse how these differ from the research perspective. To achieve this goal, we purposefully selected experienced teacher-researchers as informants in a qualitative empirical study. Given their experience as both teachers and researchers and their current positions in which they teach and conduct research concurrently, we expect these teacher-researchers to be able to explicate perspectives on quality grounded in experience. This study formulates a stipulative definition of quality concerns from the practice perspective, based on teacher-researchers’ frames of reference.

Method

Context

After several initiatives stimulating secondary school teachers to conduct PhD research, and finding that upon completion teachers left their schools to pursue other careers, the ministry of Education in the Netherlands offered subsidies to create postdoctoral positions in secondary schools for science and mathematics teachers. Teachers with a PhD could apply for a grant to spend two days a week on a postdoctoral research project for two or three years while maintaining their teaching positions in secondary education. The research projects were conceived from the beginning to contribute to the advancement of both educational practice and research. Consequently, there was a need to attend to the concerns and needs of both educational practitioners and researchers. Owing to their backgrounds and current positions, it was expected that the recipients of the grant were able to consider both the research and the practice perspectives on quality. Therefore the ten recipients of the postdoctoral grants were purposefully selected (Patton 2002) as informants for this study. In conversations with the individual teacher-researchers, the first author explained the purpose of the study and their role as informants. The teacher-researchers gave oral consent to participate in this study.

Informants

The ten teacher-researchers hold doctoral degrees in either science, mathematics, or science or mathematics education, and have between five and twenty-one years of experience as a science or mathematics teacher in secondary education. Owing to their experience as researchers, the teacher-researchers have a clear perception of what research is (cf. Laroes et al. 2018) and are familiar with the research perspective on quality; owing to their experience as teachers, they are able to provide a teacher’s perspective on quality.

The teacher-researchers’ research projects were all design-based research studies that addressed a current issue within science or mathematics education in the teacher-researchers’ schools. The research proposals were written by the teacher-researchers in close collaboration with their school leader and colleagues, and a university-based researcher, thus ensuring the embeddedness of the research project within the practice context and current research respectively. Table 2 provides an overview of the informants and their research projects.

Table 2. Overview of the teacher-researchers and their research projects

Teacher-researcher^a	School subject	Title of postdoctoral research project
Anna	Chemistry	Interrelatedness of context-based chemistry education and student needs
Alex	Biology	A practical approach to within classroom differentiation using videos
Daniel	Mathematics	Differentiation according to students’ interest in mathematics education
John	Biology	Influence of knowledge on neurological processes on teachers’ classroom practice
Leonard	Physics	Multidisciplinary science contexts for flexible use of concepts
Mark	Biology	Professional development in a teacher development team focused on design, use and evaluation of context-concept education
Mike	Biology	Use of data to enhance teachers’ teaching practices
Oscar	Physics	Modelling in physics education
Peter	Physics	Technical internships to enhance students’ motivation for science
Tessa	Chemistry	Enhancing language proficiency of grade 10 students in science education

^aAll names are pseudonyms.

Data collection

Data were collected from the teacher-researchers using individual reflections, small group discussions, and individual semi-structured interviews, leading to method triangulation (Miles and Huberman 1994). Multiple methods for data collection were used to create multiple and different opportunities for the informants to reflect on the quality of practice-oriented educational research. The individual reflections and small group discussions provided initial opportunities for the teacher-researchers to articulate their quality concerns. These articulated concerns provided input for informed interviews to gain deeper understanding of their quality concerns.

The teacher-researchers had bimonthly meetings to discuss the progress of their research projects and difficulties that they experienced. During one of these meetings, one year after starting their research projects, the teacher-researchers were asked to write an individual reflection based on three guiding questions. They were asked what they considered to be important concerning quality of practice-oriented educational research, what they thought was good about their own research and why, and what they did to ensure the quality of their own research and why they did it that way. There were no restrictions on what the teacher-researchers could write down. The individual reflection was followed by small group discussions with two or three teacher-researchers. They were asked to share their individual reflections and to discuss differences and similarities.

Thereafter, the research-focused concerns from the research perspective, as described in Table 1 (e.g., truth value, applicability, consistency, neutrality, and cumulativity) were introduced to the teacher-researchers. It was assumed that the teacher-researchers were already familiar with these quality concerns as they were experienced researchers. Therefore it was expected that the introduced concerns provided no new information, but would trigger their thinking about these quality concerns. The teacher-researchers were asked to write down in a second individual reflection parallels and differences between the research-focused concerns from the research perspective and their own perspective on quality, including any additional quality concerns from their perspective as teachers. The individual written reflections were collected, and the small group discussions were recorded and summarized.

The practice-focused quality concerns as described in Table 1 (e.g., comprehensibility, acceptability, usability, value for learning, and economic value) refer to the effects on educational practice, but are based on a research perspective. These quality concerns were assumed to be less familiar to the teacher-researchers, and it was expected that the teacher-researchers would hold different quality concerns in this domain. The practice-focused concerns were therefore not introduced to the teachers-researchers, because we wanted them to reflect on this with open minds.

The outcomes of the individual reflections and the small group discussions guided the semi-structured interviews with the individual teacher-researchers five months after the meeting. The teacher-researchers were asked clarifying questions concerning their individual reflections (e.g. what they meant by certain terms), why they considered the quality concerns in their individual reflections to be important and how they addressed them in their research. Subsequently, they were asked what they considered important concerning the quality of research of others: research of the other postdoctoral teacher-researchers and research in scientific journals. They were asked what aspects they paid attention to and why. All interviews were recorded and transcribed.

Data analysis

Data analysis was based on an informed grounded theory approach as established by Thornberg (2012). Building on the impossibility of pure inductive reasoning as advocated in the classic grounded theory approach of Glaser and Strauss (1967) and on the analytical value of abduction in addition to induction, Thornberg expands Charmaz (2014) constructivist grounded theory approach by including extant theories as sources of inspiration during the data analysis and theory building procedures. This results in a theory ‘thoroughly grounded in data by GT methods while being informed by existing research literature and theoretical frameworks’ (Thornberg 2012, 249). In this study, the research-focused quality concerns as described in Table 1 are used as sensitizing concepts (Bowen 2006) to provide a starting point for our analysis and ensuing grounded theory. During analysis and theory building, we move beyond the initial sensitizing concepts by changing and expanding them, leading to a theory grounded in the data.

To gain an overview of the variety of teacher-researchers’ quality concerns, the first step of data analysis was the open coding (Charmaz 2014) of all data using the five research-focused quality concerns as described in Table 1 as sensitizing concepts. From this initial coding it became apparent that their quality concerns often pertained to specific aspects of research instead of to research as a whole. In subsequent coding, we differentiated between concerns for research as a whole and three aspects of research i.e., intervention, method and results. The initial codes for quality concerns were sorted and clustered using the method of constant comparison. This resulted in thirteen quality concerns, among which five that were inspired by the used sensitizing concepts.

It should be noted that the teacher-researchers used various terms to describe similar quality concerns and vice versa. For example, descriptions of quality concerns pertaining to ‘consistency’ were also termed ‘imitability’, ‘reproducibility’ and ‘repeatability’. The other way around, the teacher-researchers used similar terms for different descriptions of quality concerns. For example, ’cumulativity’ was used to refer to contributing to scientific and practical knowledge, but also to refer to contributing something extra to student learning on top of the regular curriculum. To overcome this issue, it was decided to ignore the specific terms used by the teacher-researchers and to focus on their descriptions of quality concerns.

The second step was focused coding (Charmaz 2014) of all data for aspects of research and quality concerns. The aspects of research and quality concerns resulting from the first step of data analysis were used as codes. All relevant quotations were coded with both types of codes. Quotations that did not address a specific research aspect were coded as referring to research as a whole. Subsequently, each coded quotation was summarized in a descriptive statement. Table 3 shows an example of a coded quotation and the resulting descriptive statement.

Table 3. Example of coding

Quotation from personal reflection by Leonard	Step 2: Codes + descriptive statement
What I find important concerning quality is that a study is embedded in other relevant research and embedded in educational practice. A study should apply what is already known.	Aspect of research: research as a whole Quality concern: cumulativity Descriptive statement: study is embedded in scientific and practical knowledge

As a third step, all descriptive statements of each teacher-researcher were collected in a matrix (Miles, Huberman, and Saldaña 2014) to gain an overview of the results per teacher-researcher. Similar descriptive statements in a cell were merged, but multiple descriptive statements remained when they were dissimilar.

The fourth step was to combine the individual matrices of the whole group of teacher-researchers into one overview matrix (Miles, Huberman, and Saldaña 2014). Again, similar descriptive statements in a cell were merged, but multiple descriptive statements remained in one cell when they were dissimilar. All descriptive statements of the thirteen quality concerns were subjected to close scrutiny using the method of constant comparison to identify overlap between quality concerns. To ensure uniqueness of categories, the revealed overlap in meaning between quality concerns led to merging of the thirteen quality concerns into seven.

All data were checked against the seven remaining quality concerns to ensure completeness and overall coverage. All quotations fitted with the seven quality concerns. The overview of the quality concerns of the teacher-researchers was therefore considered representative of the data.

Validation of the data analysis was done using an audit procedure as developed by Akkerman et al. (2006) and extended by De Kleijn and Van Leeuwen (2018). The summative audit was performed by a peer researcher who was not involved in the study. Being a teacher-researcher herself, the auditor was sensitive to the teacher perspective in the data and mindful of (mis)interpretation from a research perspective by the research team. The auditor considered the data analysis procedure to be visible, comprehensible and acceptable, and the ensuing description of results to be representative of the data.

Results

Aspects of research

Quality concerns for practice-oriented research as expressed by the teacher-researchers typically pertained to specific aspects of research instead of to research as a whole. Aspects of research that they distinguish are intervention, method and results. For example, when asked to reflect on what they considered important concerning the quality of practice-oriented educational research, the responses of two teacher-researchers pertained to different aspects of research. Daniel, who was developing teaching materials to be used as an intervention in his study, replied:

‘Teaching materials have good/desired learning results. The developed teaching materials should be based on a solid theoretical foundation. They add something to existing education, replace or improve it.’ (Daniel, reflection, emphasis added)

John referred to the results of his study:

‘The results withstand scientific scrutiny. They yield useful insights for teachers and students and contribute to improvement of educational outcomes and teaching methods.’ (John, reflection, emphasis added)

Quality concerns

The teacher-researchers elaborated seven quality concerns for practice-oriented educational research. These concerns form the stipulative definition of the quality of practice-oriented educational research from the teacher-researchers’ perspectives. An overview of their quality concerns is displayed in Table 4. It should be noted that the order of the quality concerns does not signify the importance of certain criteria over others. Five quality concerns were named after the research-focused quality concerns as displayed in Table 1: ‘truth value’, ‘applicability’, ‘consistency’, ‘neutrality’ and ‘cumulativity’. Although the teacher-researchers did not express exactly the same concerns as discussed in the literature, the intent of their concerns is similar enough to justify the use of the same terminology. The two remaining quality concerns, ‘recognizability’ and ‘effectivity,’ did not display significant overlap with the quality concerns from the literature and were named by the research team based on the teacher-researchers’ descriptions of their quality concerns.

Table 4. Quality concerns for practice-oriented educational research according to teacher-researchers

Quality concern	Description
Truth value	Extent to which the results of a study are accurate and in correspondence with empirical reality
Applicability	Extent to which a study, intervention, method and/or result is feasible in and relevant for the context of a study and/or other contexts, in research and/or in educational practice
Consistency	Extent of clarity on the chain of reasoning and effectuation of a research study, and how all aspects of research cohere, so that a study, intervention, method and/or result can be replicated in research and/or practice
Neutrality	Extent to which a study and/or result is independent of the researcher
Cumulativity	Extent to which a study, intervention, method and/or result builds on and contributes to scientific and/or practical knowledge
Recognizability	Extent to which a study, intervention and/or result is in accordance with a teacher’s professional experience and expectations
Effectivity	Extent to which a study and/or intervention contributes to educational practice

The teacher-researchers did not create explicit contrast between research-focused and practice-focused concerns but combined both perspectives in universal quality concerns that focus on research and practice simultaneously.

Truth value

The teacher-researchers stated that results of research should be an accurate representation of occurrences in natural practice settings.

‘Research should originate in practice and be conducted in practice, but it should be scientifically underpinned (…) The analysis should be well substantiated and because of that it becomes possible to write down what the exact effects in practice are.’ (Peter, reflection)

According to the teacher-researchers, attention to method is essential to meet this concern, for example by choosing methods that fit with the complexity of real-life classroom settings or by combining quantitative and qualitative methods in ways that they reinforce each other. Quality concerns pertaining to ‘truth value’ were least mentioned by the teacher-researchers.

Applicability

Quality concerns relating to ‘applicability’ were most elaborately discussed by the teacher-researchers. They broke this concern down along two continua, as displayed in Figure 1. The first continuum is from applicability in research to applicability in practice.

‘Research should be relevant for school practice and the world of science. A good study should do these two things simultaneously. If you do research that is only meaningful for schools, that raises the question what makes it relevant for science. But scientific research that is not relevant for schools is also not desirable.’ (John, reflection)

Figure 1. Applicability along two continua

The second continuum is from applicability in the context of a study to applicability in other contexts.

‘Research should be practice-oriented in a sense that it has effect in the practice of study, but it should also yield generalisable results. Generalisable results, because other teachers should also be able to apply it. It is directly practically applicable for people who work in education.’ (Leonard, interview)

The teacher-researchers valued applicability along both continua. They stated that a study, intervention, method or result should be feasible in and relevant for other target groups, school subjects and levels of education in both research and practice. For example, the intervention, method and results from a study of in-classroom differentiation in 10^th grade biology education using instructional videos should also have relevance for 8^th grade English education, in research as well as in practice. An issue strongly emphasized by the teacher-researchers was the potential for the practical use of research, interventions, methods and results in educational practice. They stressed the importance of feasibility with teachers’ concerns, knowledge and skill levels, time constraints, and connection with school culture and policy.

‘I try to connect my research to the interests and background of the teachers I work with. That is a condition if you do practice-oriented research, mainly concerning relevance. If you do something that nobody is waiting for than people won’t appreciate it. And concerning background: teachers have to be able to understand it and use it.’ (Leonard, interview)

‘Concerning differentiation, it is most important that it is practically feasible for an average teacher with a fulltime job and classes with 25 to 30 students. One of the most important things is that it is a heuristic solution and thus a time saving procedure.’ (Alex, interview)

Compatibility with these issues is an important condition for research to find actual applicability in educational practice instead of remaining as potential applicability. To achieve compatibility with these issues, it is important that a study is grounded in real-life educational practice and that teachers are involved for example, in developing and executing an intervention.

Consistency

Consistency was also elaborately discussed by the teacher-researchers. They described it as a two-level concept. First, the chain of reasoning from research question to intervention to method to results and conclusion should be clear, and all elements of a study should be coherent. They also argued that research, intervention, method and results should be meticulously described, both the design and effectuation. The teacher-researchers wanted clarity about what was done exactly e.g., used methods, used teaching materials and the role of the researcher.

‘Research should be trackable. I should be able to see what was done and what the role of the researcher was. It should be systematically trackable.’ (Mark, interview)

‘You rarely encounter teaching materials in research, even when the research is on teaching materials. And those results are published and then I wonder (…) what actually happened.’ (Oscar, interview)

Second, a study, intervention, method or result should be replicable. According to the teacher-researchers, clarity about the chain of reasoning and effectuation are necessary conditions to achieve this.

‘In an article, it must be clear what decisions were made and why. The underpinning must be structured in a logical way. That is important for the trustworthiness and to be able to reproduce it.’ (Daniel, interview)

The teacher-researchers’ perceptions of what should be replicable varied. For example, it was variously argued that the effectuation of an intervention, data analysis, or results should be replicable. There was also diversity in perspectives on the contexts in which a study, intervention, method or result should be replicable: in the same context, similar contexts or other contexts.

Neutrality

Concerning neutrality, the teacher-researchers stated that a study, data analysis and results should be independent of the researcher. However, they perceived obstacles in achieving this neutrality in their own research. They argued that, as teacher-researchers, they have multiple roles in their studies e.g., as designers, teachers and researchers of an intervention simultaneously. Furthermore, the teachers did their research in their own school, with their own students and colleagues. According to the teacher-researchers, both obstacles could lead to bias.

‘Neutrality is difficult because I do research in my own school, and because I do the intervention myself but also study that intervention and evaluate it. So I have all these roles simultaneously. (…) It is kind of like marking your own paper, you know? (…) And people might also be a little less critical, because they do not want to criticize their colleague. (…) These kinds of elements play a role. Well, you could ignore it, but it is simply the case.’ (John, interview)

Cumulativity

The teacher-researchers expressed that a practice-oriented study and its research questions should be embedded in educational practice. Practical knowledge should be utilized in the design and effectuation of a study, intervention and method. This practical knowledge can be rooted in the teacher-researchers’ personal base of knowledge and experience, or those of their colleagues. The teacher-researchers also valued scientific knowledge in this respect, but they expressed critical concern about its utility owing to perceived lack of generalisability, the existence of contradictions and the possible impediment of originality.

‘In the 20 to 25 years that I am teaching I have heard so many sure theories. (…) Look, it is not physics, there are no laws in our education. So I could say if you do this, chances are this, or if you do that, chances are a little bit bigger, but I can’t say for certain what will happen. (…) So I could say, ’I am going to build on that theory’, but they are basically just frames of reference. You have to really realize that.’ (Alex, interview)

It was also argued by the teacher-researchers that practice-oriented educational research offers opportunities for practical knowledge to become embedded in scientific knowledge, but also for theoretical knowledge to find its way into teachers’ practical knowledge. They stated that the combination of practical and scientific knowledge can elevate the quality of a study, intervention, method or result.

‘Colleagues have of course a lot of knowledge and skills because they have been teaching for years, so [they have] a sort of practical knowledge and experience from which they can extract more general statements or characteristics. Those could become a sort of ‘working hypothesis’ if you would formulate them a little more abstractly. Then you start generalising and then you can test whether something indeed is the case. And that way you get higher level knowledge, knowledge that is independent of a specific educational practice. Then it becomes scientific knowledge.’ (Leonard, interview)

Recognisability

According to the teacher-researchers, research should resonate with a teacher’s personal, professional perception of educational practice, students and teaching. It should treat issues that a teacher considers important or difficult to be deemed relevant. Furthermore, it should be possible for a teacher to imagine how an intervention or research result would work in their own educational practice. Unlike the other quality concerns, this concern is centred around an individual teacher’s perspective. All teachers have their individual experiences and expectations, and from this personal frame of reference they consider research. If a teacher recognizes a study as being in accordance with their frame of reference, they will be more open to using an intervention or results from research. However, when a study introduces an intervention or result that seems counterintuitive to a teacher’s experiences or expectations, they will be more inclined to ignore it.

‘I pay attention to a lot of things [when reading an article], but what I immediately do, that is kind of a reflex, is to think: Is it practical? How do I do it? How should I do it? How can I use this? Well, furthermore you are constantly comparing with your own, with reality. But that is of course because you are teaching. I am constantly looking: Do I recognise myself in this?’ (Alex, interview)

Effectivity

A second quality concern introduced by the teacher-researchers was ‘effectivity’. The teacher-researchers emphasised this concern and were very clear about this issue: a practice-oriented educational research study should contribute to educational practice. They argued that a study or intervention should improve educational practice by optimizing teaching and learning processes for example, by creating new teaching materials. This should have positive effects on student learning or motivation, or contribute to growing understanding of classroom interaction, teaching or student learning by the teachers involved.

‘My aim is to reach as many students as possible. My research is the best it can be when this aim is achieved. (…) What is good about my research is that it leads to growing insight of myself and my colleagues in teaching and learning, and that it leads to teaching material that improves student learning.’ (Oscar, reflection)

Conclusions and discussion

The aim of this study was to identify teachers’ perspectives on the quality of practice-oriented educational research and to analyse how these differ from the research perspective. In a qualitative study, teacher-researchers were questioned about their perspectives on the quality of practice-oriented educational research. The results of this study display three notable features of teacher-researchers’ perspectives.

First, the teacher-researchers do not explicitly distinguish between research- and practice-focused quality concerns, in contrast to quality concerns in the literature. From this lack of explicit contrast between research- and practice focused quality concerns, it can be contended that the teacher-researchers have similar concerns about quality for research and practice. This seems to reflect the equal importance of the two-sided purpose of practice-oriented educational research namely, to contribute to educational research and practice. The lack of explicit contrast between research- and practice-focused concerns could reflect the two-sided perspectives of the informants as both researchers and teachers. However, it could also be an artefact of our data analysis in which we integrated some concerns for research and practice owing to conceptual alignment between concerns for research and practice. This could also explain why some quality concerns occur in our results more often than others. The quality concern ‘applicability’ occurs most often possibly because of the broad meaning we assigned to it, owing to conceptual alignment between concerns for applicability in research and practice. On the other hand, the quality concern ‘truth value’ occurs least often possibly because of the more narrow meaning we assigned to it. “Truth value’ is a strictly research-focused concern focussed on correspondence with empirical reality which we separated from the more practice-focused concern for ‘recognisability’ focussed on teachers’ perceived reality, owing to lack of conceptual alignment between the two concerns.

Second, the teacher-researchers tend to focus on different aspects of research instead of only on research as a whole when discussing the quality of research. This differentiation between different aspects of research could result from the teacher-researchers’ preoccupation with interventions, methods and results in their own research. Interventions play a central role in the design-based research studies of the teacher-researchers in which they design and test an intervention. Their focus on interventions might also stem from their perspectives as teachers and their ensuing interest in the content, implementation and effects of an intervention. This aligns with one of the concerns that teachers express about educational research in general. According to teachers, research should indicate how it can be used in educational practice (Gore and Gitlin 2004; Ratcliffe et al. 2005). Concerns about method might emanate from their perspectives as researchers. From this perspective, they are interested in how a study was executed to assess the quality of research and to acquire ideas on methods for their own research. Concerns about results might originate from both their practice and their research perspectives. The teacher-researchers are concerned about results because they want to know what a practice-oriented educational research study (potentially) contributes to educational research and practice.

Third, seven concerns for the quality of practice-oriented educational research result from our study. These quality concerns overlap with quality concerns expressed by researchers in the literature (Table 1), but they also expand, complement and constrain them. The teacher-researchers’ quality concerns related to ‘truth value’ are similar to concerns about this issue in the literature. Both express that results should be an accurate representation of occurrences in natural practice settings. Quality concerns about ‘applicability’, ‘consistency’, ‘neutrality’ and ‘cumulativity’ are expanded in their meaning. Concerns about ‘recognizability’ and ‘effectivity’ are introduced by the teacher-researchers. Concerns about ‘comprehensibility’, ‘acceptability’, ‘usability’, ‘value for learning’ and ‘economic value’ as expressed in the literature are either integrated in other quality concerns or are not mentioned by the teacher-researchers.

Expanded concerns

The teacher-researchers’ quality concerns about ‘applicability’, ‘consistency’, ‘neutrality’ and ‘cumulativity’ display similarities as well as differences with researchers’ quality concerns in the literature.

First, the teacher-researchers’ descriptions of ‘applicability’, ‘consistency’ and ‘cumulativity’ are more specific by pointing out which aspects of research should be applicable, consistent and cumulative e.g., intervention, method, result. Additionally, where ‘applicability’ is described by researchers as the extent of generalizability, the teacher-researchers make this more specific by describing it as feasibility and relevance.

Second, they expand the descriptions of ‘applicability’, ‘consistency’ and ‘cumulativity’ by explicitly referring to educational practice in addition to educational research. The teacher-researchers point out that research studies should be applicable and replicable not only in other context for research, but also in educational practice so that schools, teachers and students can benefit. Concerning ‘cumulativity’, the teacher-researchers are convinced of the value of practical knowledge in addition to scientific knowledge.

Third, the teacher-researchers’ descriptions of their concerns about ‘consistency’ and ‘neutrality’ differ from researchers’ descriptions. However, these different descriptions are not contradictory but congruous.

Complementary concerns

In the complementary quality concerns introduced by the teacher-researchers, their teacher voices shine through. ‘Recognisability’ seems to be the practice-focused variant of the more research-focused ‘truth value’. By addressing this quality concern, the teacher-researchers bring into focus the importance of a teacher’s perceived truth in addition to empirical truth. This quality concern aligns with Bartels (2003) assertion that teachers establish the quality of research on an individual level, in contrast to researchers who establish the quality of research on a community level. This fits with the personal nature of teaching in contrast to the more impersonal nature of research (McIntyre 2005). It is also in accordance with teachers’ concerns about educational research in general i.e., that research findings should resonate with teachers’ professional experience (Everton, Galton, and Pell 2000; Labaree 2003; Ratcliffe et al. 2005) and apply to their professional contexts (Gore and Gitlin 2004; Vanderlinde and van Braak 2010).

In ‘effectivity’, the teacher-researchers’ teacher voices shine through in their focus on (the potential for) change in educational practice. The teachers-researchers want to see the effects of research in the context of a study. Absence of these effects seems to mark research as infeasible for and irrelevant to their own educational practice. In this respect, ‘effectivity’ can be considered a prerequisite for concerns related to applicability in educational practice.

The introduction of ‘recognisability’ as a quality concern highlights the importance of teachers’ individual frames of reference in assessing the quality of practice-oriented educational research. This may explain why teachers are hesitant to use research and its results. As already established, teachers and researchers have differing reference frames and view research from differing perspectives (e.g., Bartels 2003; Hammersley and Gomm 2002; Joram 2007; Ratcliffe et al. 2005). It is plausible that this creates a mismatch between the way that educational practice is portrayed by educational researchers and the way that it is perceived by teachers. This may lead teachers to ignore research, because people are generally inclined to ignore information that does not fit with their preconceived notions and frames of reference (Kahneman 2011).

Quality concerns related to effectivity provide another explanation for the limited use of research and its results by teachers. For teachers, the effects of implementing an intervention or result from research in their educational practice should be in proportion with the required effort (Doyle and Ponder 1977). This implies that research should display sufficient effectivity for teachers to consider making an effort to use it in educational practice.

Constrained concerns

Researchers’ practice-focused quality concerns do not emerge as separate concerns in the teacher-researchers’ perspectives. Concerns about ‘comprehensibility’, ‘acceptability’, ‘usability’ and ‘value for learning’ are partly integrated into teachers-researchers’ concerns about ‘applicability’, ‘recognisability’ and ‘effectivity’; concerns on ‘economic value’ were not addressed and are therefore considered beyond the scope of the teacher-researchers’ concerns.

Limitations

The established stipulative definition of quality concerns for practice-oriented educational research is based on teacher-researchers’ perceptions. The respondents in this study were in rather distinctive positions as teachers in secondary education while simultaneously holding a postdoctoral research position at a university. It is assumed that the respondents’ perceptions of quality of practice-oriented educational research are based on their experiences as both researchers and teachers. However, it is unknown whether these perceptions are in accordance with the perceptions of teachers without research experience or in other sectors of education. Additionally, the stipulative definition does not provide directives for how to address the quality concerns in practice-oriented educational research studies. These issues remain for further research.

Implications

The used informed grounded theory approach (Thornberg 2012) was a good fit for this study. Given the topic, a classic grounded theory study purely based on inductive reasoning would not be possible since we as educational researchers already had knowledge of and ideas about quality concerns for (practice-oriented) educational research. We also considered it undesirable to ignore existing knowledge and theories present in the literature. The informed grounded theory approach as established by Thornberg (2012) provided the opportunity to use pre-existing ideas and theories in a non-constraining way by complementing inductive analysis with abductive reasoning inspired by this pre-existing knowledge. This resulted in a new theory grounded in the data without ignoring extant knowledge and without imposing predefined categories on the data. Since this approach is rooted in constructivist grounded theory, it should be recognized that the codes and categories used in the data-analysis and the resulting grounded theory did not emerge from the data, but that they are the product of deliberate interpretation by the researchers (Charmaz 2014).

The stipulative definition of quality for practice-oriented educational research from teacher-researchers’ perspectives identified in this empirical study differs from researchers’ stipulative definition based on the educational research literature. Even though the differences are small, they are meaningful. Nevertheless, it can be argued that, instead of emphasizing these differences, emphasis should be on researchers’ and teachers’ common concerns. By emphasizing their common quality concerns, researchers and teachers can work together to address the challenges of legitimacy and relevance faced by research in practice. In addition, it can be argued that their different definitions reflect the substantial differences between research and teaching. Labaree (2003) argues that it should not be attempted to eliminate the differences between teachers’ and researchers’ perspectives. Moreover, they can easily coexist since they do not exclude each other and as long as neither of them claims legitimacy over the other (Hammersley 2007; Wardekker 2000). Considering both arguments, we suggest that teachers should become more involved in research. So far, teachers are already included in, for example, discussions on which research topics to address or in development and execution of interventions. We suggest to involve them in more substantive ways by also including them in the development of research proposals, in decision making on research funding and in the execution of research projects. Taking their common quality concerns as a starting point, close collaboration between both stakeholders provides teachers with the opportunity to voice their divergent concerns. It simultaneously provides researchers with the opportunity to address teachers’ concerns in all phases of a research project. This could decrease researchers’ challenges concerning legitimacy and relevance of their work and increase teachers’ use of research in educational practice, resulting in a more evidence-based educational practice.

As established by Verschuren (2009, 13), sceptics of practice-oriented research ‘believe that criteria for practice-oriented research are easier to fulfil than those for theory-oriented research. However, […] the opposite is true; it must fulfil more and more complex criteria than theory-oriented research’. The quality concerns based on the two-sided research and practice perspective of teacher-researchers are more complex than the quality concerns based on the research perspective in the educational literature. This study highlights how the inclusion of teachers’ individual frames of reference in the stipulative definition makes addressing these quality concerns more complex. In particular, addressing quality concerns about ‘recognisability’ provides a serious challenge for practice-oriented educational researchers. The personal character of this quality concern means that there is no guarantee that teachers will use a research study and its results, even if the other quality concerns are met. This is not a flaw on behalf of teachers or of the stipulative definition; it emphasizes the subjective nature of quality and the importance of teachers’ professional judgement.

Acknowledgments

The authors thank Floor Kamphorst for her thorough audit of our data analysis procedure.

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

This work was supported by the Dutch Ministry of Education, Culture and Science under grant number 804AO-42742.

Notes on contributors

S. E. A. Groothuijsen

S. E. A. Groothuijsen is a PhD candidate at the Freudenthal Institute for Science and Mathematics Education at Utrecht University in the Netherlands. Her research focusses on the quality and impact of practice-oriented educational research.

L. H. Bronkhorst

L. H. Bronkhorstworks as assistant professor at the Department of Education of Utrecht University. Her research focusses on learning, development and collaboration across contexts and she teaches different courses in the academic, research and teacher education master program.

G. T. Prins

G. T. Prins is an Assistant Professor in science education at the Freudenthal Institute for Science and Mathematics Education at Utrecht University in the Netherlands. His research focusses on education for sustainability in chemistry by means of design based research. He is involved in several pre- and in-service teacher training programs, as well as governmental funded projects in which science teachers combine teaching tasks with conducting practice-oriented educational research.

W. Kuiper

W. Kuiper is Emeritus Professor of Curriculum Evaluation at the Freudenthal Institute for Science and Mathematics Education at Utrecht University in the Netherlands. From 2008 till 2018 he was the head of the Curriculum Research Department at the Netherlands Institute for Curriculum Development (SLO).

Notes

1. For example, Anderson and Herr (1999) and Heikinnen, de Jong, and Vanderlinde (2016) elaborate quality criteria for practitioner research that can be regarded as practice-oriented educational research executed by teachers. However, owing to the focus of their criteria on the procedures of research, they do not address the quality of a research study in itself.