Investigating ‘Presence in teaching’: explicating the transition from qualitative studies to a survey instrument

ABSTRACT

This paper provides a detailed account of the development of an instrument to investigate the emerging concept of presence in teaching (PiT) on a larger scale, explaining how the transition from data and findings from qualitative studies to a survey instrument was accomplished. In order to ensure relevance for teachers, the instrument needed to do justice to the unique and varied character of the qualitative data and be closely related to educational practice. Therefore, we adopted an interpretivist approach and a procedure involving different types of interactions: between outsiders (researchers, experts) and the instrument, between insiders (teachers) and the instrument, and between outsiders and insiders. Often, instruments are developed from an outsider perspective, in which respondents serve as informants for the researchers. Our development process occurred in a dialectical relationship between the ‘scientific’ concept and the ‘everyday’ concept (in the intuitive understanding of teachers) of PiT, which allowed us to capture PiT within a survey by means of narrative vignettes using the language and classroom practices of teachers. Reflections on the quality criteria and methods used, the challenges encountered and solutions found, may serve educational researchers planning to investigate emerging concepts such as PiT on a large scale.

KEYWORDS:

• Measuring emerging concepts
• presence in teaching
• interpretivist perspective
• mixed methods in educational research
• insider-outsider perspectives

Introduction

This paper delves into a specific phase in an extensive study on presence in teaching (PiT): the transition from qualitative in-depth studies to the development of a quantitative research instrument. Such a mixed-methods way of working is generally used within educational research in order to generalize or extend the initial qualitative, exploratory findings (Creswell and Plano Clark 2011). However, the considerations and methodological procedures used to make such a transition are seldom explained in research articles (cf. Alise and Teddlie 2010) or in methodological literature (cf. Onwuegbuzie et al. 2010; Maxwell 2016). Nevertheless, there are several challenges associated with this transition phase, such as how to use the qualitative results that inform the instrument development (Creswell and Plano Clark 2011), which quality criteria to use and how to justify the decisions involved in the transition process (Shannon-Baker 2016). In this article, we address exactly this transition phase, by presenting an elaborated account of the development of an instrument to investigate PiT on a large scale, informed by qualitative findings and data.

Presence refers to being completely immersed in the here and now (Scharmer 2007), and perceiving and acting with all one’s being and all one’s senses, not merely rationally (Noddings 2013). Presence in teaching (PiT) concerns the manifestation of presence within the daily practice of teaching and refers to the teacher’s experience of being in a state of full attentiveness, awareness and responsiveness to what is happening in the moment in class (Rodgers and Raider-Roth 2006; Roefs et al. 2021b). Both presence and PiT are emerging concepts in the context of education. Rodgers and Raider-Roth (2006) developed a conceptualization of PiT that was built on (educational) philosophy, psychology, religion and art. They also drew upon data addressing the notion of presence from interviews, group discussions and (student) teachers’ papers. In their conceptualization, PiT consists of three dimensions: connection to self, connection to students and pedagogical connection. The last refers to the teacher’s actions embedded in the situated interactions between teacher, students and subject matter. They considered PiT particularly relevant to students’ broad development. There are a few qualitative studies available on PiT, all among small samples of purposefully selected (predominantly secondary) teachers (Solloway 2000; Meijer et al. 2009; Roefs et al. 2021b). These studies yielded in-depth understanding and rich detail about aspects of PiT. Solloway (2000) found that when teachers were being present, they experienced receptiveness and non-judgmental attention to students. In a study by Meijer et al. (2009, 298), a single student teacher supported by supervision sessions developed presence, which was regarded as ‘being-while-teaching’, connecting personal and professional aspects of teaching. In their qualitative study, Roefs et al. (2021b) found three aspects of PiT: self-awareness, focus on students’ learning and experiences and acting responsively. These aspects corroborated the three dimensions of PiT in Rodgers and Raider-Roth’s (2006) conceptualization. The analysis by Roefs et al. (2021b) specified that teachers experienced a natural in-the-moment focus on students; when they were simultaneously self-aware, they experienced full-fledged presence, which was considered important for acting responsively. These results indicated that PiT involves three simultaneously occurring aspects. Additionally, the findings from that study confirmed the intersubjective character of PiT, following from the dependence of PiT on the interactional process between teacher, students and subject matter. These two characteristics – simultaneity and intersubjectivity – were implicit in the conceptualization of PiT (Rodgers and Raider-Roth 2006) and in other qualitative studies into PiT (Solloway 2000, Meijer et al. 2009). A study among secondary school students of their experiences of presence in class revealed aspects of teaching that enabled them to be present (Roefs et al. 2021a): recognition, listening and the encouragement of students’ participatory interaction with the subject matter; these aspects corresponded to PiT.

Additionally, many variations in teachers’ experiences of PiT were found that could be associated with the nature of the situational classroom context, personal characteristics, teaching subject or the school’s educational approach (Solloway 2000, Roefs et al. 2021b). Considering the small sample sizes, these relations were not systematically investigated further. The meaning teachers attached to presence concerned aspects of students’ broad development and their own feelings of fulfilment on several levels (Roefs et al. 2021b).

Since earlier studies were conducted among small groups of teachers with an interest in presence, it is not known whether PiT has a broader resonance among a large, random sample of teachers. Following up on these qualitative results, we aimed to measure the extent to which a large and diverse group of secondary teachers (most of whom are likely not familiar with the concept of PiT) experiences PiT in their daily practice and associate particular meanings with PiT. Additionally, we aimed to further explore variations of PiT and their possible relations with respondent-specific characteristics. To investigate experiences and meanings among a large sample, a survey was deemed an appropriate instrument.

Presenting a meta-framework for instrument development using mixed methods, Onwuegbuzie, Bustamante, and Nelson (2010) argued that the items need to be sensitive to the diversities in the target population. Moreover, we wanted to develop an instrument that was considered relevant by teachers. This is important because educational research is criticized by teachers for being far removed from their daily practice and therefore not always relevant to teaching (Drill, Miller, and Behrstock-Sherratt 2013). In order to ensure relevance, the survey instrument to be developed needed to do justice to the unique and varied character of the qualitative data and findings and be closely related to educational practice. An interpretivist approach to instrument development seemed most appropriate to cater for these two features. From an interpretivist perspective, reality is considered socially constructed. What is experienced and deemed meaningful depends on the situation as captured at a particular moment of time (Angen 2000). By negotiating multiple ways of understanding PiT within the development process, we assumed that we could remain close to educational practice, while also including the richness of existing qualitative data and findings. An interpretivist stance was also preferable insofar as we did not begin from pre-determined hypotheses, but maintained an attitude of openness to teachers’ experiences and views (cf. McChesney and Aldridge 2019).

With this study, we aim to contribute to methodological knowledge of the significance and application of mixed methods, by broadening the discourse with an elaborated account of the development of an instrument to investigate PiT quantitatively. Following the call from mixed-methods methodologists to indicate paradigmatic underpinnings (e.g. Alise and Teddlie 2010, Creswell and Plano Clark 2011, Shannon-Baker 2016), we will explicate how the interpretivist perspective guided the development process as well as the design of the final instrument. As we could not follow existing procedures or foresee problems we might encounter, the following methodological question was formulated: How can a survey instrument for investigating PiT on a large scale be developed, doing justice to the unique and varied character of the qualitative data and findings and closely related to educational practice?

In this article, we first outline the selected quality criteria and methods for instrument development, and how we intended to apply them. Second, the actual instrument development process is described: the steps taken, with the challenges and complexities encountered and the solutions found. Finally, we draw conclusions and discuss our survey instrument as well as the development process and we reflect on the benefits and limitations in light of our methodological question.

Quality criteria and methods

The quality criteria for instrument development were selected from methodological literature on mixed methods for instrument development (Onwuegbuzie et al. 2010) and compared to literature on validity (Long 2017) and validation (Dellinger and Leech 2007) in mixed-methods research. We selected the criteria that discuss issues of meaning interpretation and argument and concern the process, in order to emphasize that establishing and demonstrating quality would be an on-going part of the instrument development. Furthermore, because little to no theory or reference instruments were available for the emerging concept of PiT, it would not have been possible to establish validity with quantitative tests.

The first criterion is: ‘Balancing insider-outsider perspectives’ (Dellinger and Leech 2007, Onwuegbuzie et al. 2010), which was considered particularly significant for using the richness of the qualitative findings and data and for the relatedness of the instrument to educational practice. It calls for a process of reaching understanding among participants, experts and researchers, which involves expressing implicit understandings (cf. Long 2017). We expected that by negotiating ‘multiple ways of seeing and hearing, multiple ways of making sense of the social world, and multiple standpoints on what is important and to be valued […]’ (Greene 2008, 20), we would be able to incorporate and balance the researchers’, experts’ (outsider) and teachers’ (insider) views. Second, for each group of stakeholders, sub-criteria concerning the quality of the instrument development were determined, drawing from the scarce literature on mixed-methods interpretivism and interpretivism in general (Table 1 provides an overview of the criteria and sub-criteria). ‘Researcher reflection’ was selected as the second quality criterion, since the process and outcome of the instrument development would depend fundamentally on how the interactions evolved, which involves learning along the way (cf. Morse 2010, Onwuegbuzie et al. 2010).

Table 1. Quality criteria and methods for development and validation of an instrument for investigating PiT on a large scale.

Quality Criteria	Perspective/Methodological Source	Description & Underpinning	Application of Methods
1. Balancing insider-outsider perspectives	MMR* MMR from the perspective of Habermas’s theory of communicative actions	Balancing insider-outsider perspectives is considered crucial to instrument development and validation using a mixed-methods approach (Onwuegbuzie et al. 2010). This overarching criterion means balancing both insider (or emic, from within the group of teachers or experts) and outsider (or etic, from the perspective of the researchers-observers) views through dialogue and debate (Long 2017).	A fine-tuned conceptualization of PiT and identification of significant statements in the qualitative data, displayed in a table of specifications linking the outsider (conceptualization) and insider (qualitative data: significant statement) perspectives, as recommended by Onwuegbuzie et al. (2010).
a. Respondent validation	Mixed-methods interpretivism Interpretivist inquiry	Respondent validation means actively engaging stakeholder participation through the principles of inclusion of ‘all relevant voices’ and dialogue (Howe 2004) in a cooperative approach (Angen 2000). This aims to contribute to capturing the (experiences of the) phenomenon in people’s ‘own terms, in their own social settings’ (Howe 2004, 54) and to co-construction of research knowledge. It has a critical element by ‘bringing expert knowledge to bear and subjecting the views and self-understandings of research participants to rational scrutiny’ (Howe 2004, 55). Torrance (2012) argued for respondent validation as a core element in MMR.	Collaboration with various teachers in the development of the instrument, in which teachers are already or will be familiarized with the conceptualization of PiT. Subsequently, qualitative pilots with teachers, with ample attention to a cognitive interview about their experiences and considerations while working with the instrument, followed by a dialogue about interpretations and improvements.
b. Expert validation	Post-positivism Interpretivist inquiry	Expert validation is the ‘systematic process of choosing, orienting, and using content experts in the judgment-quantification stage of instrument development’, contributing to content validation (Grant and Davis 1997, 269). It consists of discussing and negotiating difference and ambiguity (Angen 2000).	Written and oral reviews of the instrument by experts, followed by dialogues. The experts will be familiarized with the conceptual underpinnings of the instrument.
c. Researcher intersubjectivity	Interpretivist inquiry MMR	A process of argumentative intersubjectivity involves discussion and argumentation between the researchers in order to consciously consider and make decisions for the next steps (Morse 2010).	Meetings for researcher intersubjectivity with the other authors, scheduled every 6 weeks for discussions of a fine-tuned conceptualization, (initial) instrument, findings from the pilots and each researcher’s interpretation (Angen 2000) and the potential consequences within the development process.
2. Researcher reflection	Interpretivist inquiry MMR	Researcher reflection means engaging ‘in an explicit, self-aware meta-analysis of the research process’ at both a personal and methodological level (Finlay 2002, 531) and with regard to how understandings evolve in the research act (Angen 2000). Constant reflection encourages the researcher to evaluate and learn from each step in the process of mixed-methods research (Morse 2010), specifically as a tool for developing instruments (Onwuegbuzie et al. 2010).	Researcher journaling, in order to describe each step in the conceptual and instrument development, with decisions, struggles, considerations; findings from meetings; (possible) consequences for new steps; and critical reflection on own understandings and pre-conceptions while operationalizing PiT, especially in relation to the insider perspective of the teacher-participants.

*Note: MMR refers to mixed-methods research.

Subsequently, methods that are common for instrument development, such as conducting pilot studies and expert reviews, were selected (Onwuegbuzie et al. 2010). We did not apply these as ‘merely tools’, but rather as ‘forms of performative, interpretive practice’ (Denzin 2010, 420). This is to say that negotiation and dialogue (cf. Howe 2004, Torrance 2012, Long 2017) were central to all of the selected methods. Table 1 (4th column) provides a description of how we intended to apply the methods.

Instrument development

This section describes the entire development process as it was finally conducted (Figure 1 provides a visualization of this process). Even though the instrument development involved an iterative process, in the following paragraphs the stages will be described separately for clarity reasons, referring to the movements among the stages. We will successively discuss: fine-tuning the conceptualization of PiT; developing the initial instrument; and piloting, reviewing and revising the instrument. The present study was reviewed and approved by Radboud Teachers Academy’s ethics approval committee under approval number 20U.015679.

Figure 1. Visualization of the process of developing a survey instrument for investigating PiT.

Fine-tuning the conceptualization of PiT

Following a common mixed-methods approach, we used the findings and data from our earlier study on teachers’ experiences of presence (i.e. insider perspectives; Roefs et al 2021b) as a primary source (Onwuegbuzie et al. 2010, Creswell and Plano Clark 2011). For that earlier study, 12 teachers were interviewed twice, in order to explore in depth their concrete experiences of presence, as ‘lived through’ in daily practice. The study had an open-ended approach: the extant literature on PiT was used to inform our interpretive understandings only in the final stage of data analysis (cf. Vagle 2014). The study encompassed a broader scope of presence within the teacher–students–subject matter interaction. An important result was the theme ‘Full attentiveness and awareness’ – encompassing three aspects: ‘Self-awareness’, ‘Focus on students’ learning and experiences’ and ‘Acting responsively’, which resembled Rodgers and Raider-Roth’s (2006) three dimensions of connection within their conceptualization of PiT (Roefs et al. 2021b). To stay close to the qualitative findings, the three aspects were used to build the constructs concerning teachers’ experiences of PiT (see Figure 2). The intersubjective character of PiT and the simultaneity of the three PiT constructs (the aspects from our study) were two unique aspects from the findings that we intended to integrate in the survey instrument.

Figure 2. Fine-tuned conceptualization of PiT from qualitative results from which the survey instrument was developed.

The earlier qualitative study on teachers’ experiences of presence (Roefs et al. 2021b) revealed several variants, for example, emphasis on the conscious or the intuitive character of teachers’ attentiveness and awareness. However, these variants were not systematically analysed. Going back to the original transcripts from this qualitative study, the three aspects were fine-tuned in order to uncover variants of Self-awareness, Focus on students’ learning and experiences and Acting responsively. Subsequently, these variants were related to extant literature on PiT and more broadly on (reflective) teaching (i.e. outsider perspectives) and compared with findings and data referring to the teachers’ role from our study on students’ experiences of presence in class (i.e. insider perspectives; Roefs et al. 2021a). First, this process resulted in a definition of PiT that includes its intersubjective character and the simultaneity of the constructs. We defined PiT as:

Experiencing that one is in a state of full attentiveness to and awareness of what is happening in the moment in class, in the course of which responses emerge from awareness of the situation and of the opportunities within it for students’ development. It is an experience of (re-)connectedness between teacher, students and subject matter.

Second, this process contributed to the development of labels and operational definitions of the variants. For example, the notion of ‘being grounded’ – drawn from a conceptualization of ‘embodied presence’ in education (Brown et al. 2016) – helped us to explicitly distinguish between a grounded and a reflective variant of Self-awareness. Additionally, we shortened the label of the second construct to ‘Focus on students’, reflecting the three variants that we found, which differed in the respective orientation of the teacher’s attentiveness and awareness towards students’ learning, students’ experiences, and students’ unique meaning making. For Acting responsively, we selected the particular responses that had been shown to contribute to students’ experiences of presence: Encouraging student participation, Recognizing students as persons possessing unique traits and abilities and Encouraging students’ interest by linking the subject matter to their minds, hearts and abilities. The result was a conceptualization of PiT consisting of three constructs and their variants that fine-tuned the original aspects of PiT from our earlier study (Appendix A presents examples of how we related the outsider and insider perspectives).

Concerning the meanings of PiT, we did not begin with any coherent constructs. The meanings derived from the qualitative studies (Roefs et al. 2021a, 2021b) were related to presence. Since we could not presuppose their relevance in relation to PiT, all meanings derived from the qualitative studies were included and labelled to serve as indicators, even if a particular meaning was mentioned only once. Criteria for the selection of indicators based on theory were not at hand and thus could not be applied in advance (cf. Meijer et al. 2001; see Appendix B for examples of how we related the outsider and insider perspectives in developing indicators of the meanings of PiT).

Developing the initial instrument: a survey using narrative vignettes

We intended to develop a survey that would do justice to the unique and varied character of the data and would be closely related to educational practice. Ultimately, the survey we developed consisted of six vignettes as the core elements. Vignettes are brief, contextualized descriptions providing snapshots of a professional at work (Miles 1990). The main reason to use vignettes was that ‘whole’ PiT experiences could be included, representing a structured combination of constructs (simultaneity) and their variants (Atzmüller and Steiner 2010), embedded in a concrete, realistic context of teacher–students–subject matter interaction (intersubjectivity). In order to represent teachers’ daily practice authentically, narrative vignettes – drawn from the qualitative data – were used in the survey. They have a ‘moment-to-moment style of description [that] gives the reader a sense of being there in the scene’ (Erickson 1986, 150, italics in the original). We expected that this could stimulate respondents to reconstruct and reflect on their own experience of PiT.

Systematic design and selection of data for the construction of vignettes

Drawing from an experimental approach to vignette construction from Evans et al. (2015), we developed a structured design for the vignettes in order to systematically select detailed descriptions of teachers’ PiT experiences from the qualitative data (Roefs et al. 2021b). According to this design, the vignettes should consist of (a) experimental aspects, which were a specific combination of variants of the three constructs; (b) controlled aspects for which we included only those experiences of PiT that were situated in plenary classroom instruction or interaction, in order to eliminate extraneous variance, and (c) contextual aspects that could demonstrate variance to provide authenticity, but which we expected would not affect the response. Concerning the experimental aspects, each vignette included a particular combination of the different variants of Self-awareness (SA, two variants), Focus on students (FS, three variants) and Acting responsively (AR, three variants), giving 2 × 3 × 3 = 18 possible combinations for the vignettes. Consequently, the variant types of PiT experiences would be similarly apportioned across all vignettes within the survey (Evans et al. 2015, 162). In accordance with this structured design, we selected a total of 31 experiences from the data.

Collaborating with teachers in vignette construction

In order to capture PiT in language and classroom practices that would be recognizable for teachers in the target population (Angelides and Gibbs 2006), we collaborated closely with teachers who varied in their acquaintance and affinity with PiT, teaching subject, teaching experience and the educational approach of their schools. The objective was to construct 18 vignettes from the 31 selected experiences of PiT. To start with, the first author cooperated with two teachers. They were familiar with the concept of PiT; thus, we expected them to be able to select experiences that gave a clear and authentic representation of PiT in daily practice. All selected PiT experiences individually, after which discussion followed. The teachers provided convincing arguments and the selection was mainly determined by them; the first author only monitored the variation in contextual situations between the vignettes. After agreement, 18 draft vignettes were constructed.

For the next step, we selected teachers who were not familiar with the concept of PiT, most likely representing the target population. The objective was to improve the vignettes step by step by collaborating with these teachers successively, until saturation was reached. Six teachers were involved: two of them were selected from our network, the others via snowball sampling. After a briefing about the conceptualization of PiT, they were explicitly invited to be critical of whether each vignette was an authentic and realistic representation of the particular combination of PiT constructs and of classroom practice (Howe 2004). Negotiating their interpretations of the vignettes yielded several new understandings and revisions to the vignettes. First, the PiT constructs were sometimes not recognizable enough in the vignettes. Additionally, teachers did not always recognize the situational contexts, for example, because they were subject-matter related. In a process that involved going back to the original data, co-reflection (on teachers’ own PiT-experiences), and co-development, the experimental aspects (variants of the constructs) were further specified, while the contextual aspects were made less specific, to leave room for several interpretations (see Table 2 for examples of adjustments made by collaborating with teachers). Second, ‘Recognizing students’ (variant of Acting responsively) was not a separate ‘act’ in daily practice according to several teachers, but was interwoven into their broad attention to students and their responsive acting. In addition, ‘Encouraging students’ interest by linking the subject matter to their minds, hearts and abilities’ (variant of Acting responsively) did not cover the vignettes in which students were encouraged to make connections themselves. These findings were described in the research journal and extensively discussed within the research team. We made an effort to think critically about the conceptualization again. Accordingly, one variant was removed and the other renamed, leaving 12 vignettes (2SE x 3FS x 2AR; see Figure 3, second framework for the modified conceptualization of PiT). Third, two teachers indicated that the vignettes represented an ideal image of teaching, because experiences of PiT in difficult situations were not included. To address this, three experiences from our qualitative study (Roefs et al. 2021b) representing complex classroom situations were selected collaboratively to construct new vignettes. In all cases, we took teachers’ insider perspectives very seriously, to ensure relevance to daily educational practice.

Figure 3. Development of conceptualization of PiT.

Table 2. Examples of vignette construction from experiences of PiT through collaboration with teachers.*

Draft vignettes based on selected experiences of PiT from qualitative data, created in collaboration with two teachers	Interpretations and choices made for vignette construction, in collaboration with six teachers	Final vignettes
#P1, Line 161:’Many of the students wanted to say something and it turned into a class discussion. And when one student said something about food shortages, I saw that the rest of them were all ears [FS2]. They considered the consequences together.’ […] Line 233: ‘Then, they came up with subjects such as ‘anarchy’ and ‘solidarity’ and really started debating. For example, if there is no food would you go over certain frontiers? This does not relate to the subject matter, but they came up with it: Are we going to fight to the death [FS3]? I also really noticed emotions, intense sometimes [FS2]. Line 237: ‘I only posed the occasional question to add some depth [AR1]. This led to something other than what I had thought up, but I saw roles developing, they followed their interests and started to see how history is part of their lives [FS2]. I thought that was more important than the facts they have to learn [SE2] and I was really enjoying it’ [SE1].	Contextual aspects. Situational context unclear. Therefore. the situation was explicitly described in the vignette; subject-related information was generalized. Self-awareness. Different interpretations about the most authentic aspect: teacher’s thoughts [SE2] or the enjoyment [SE1]. Agreement reached: thoughts seemed to be a justification afterwards, whereas the enjoyment is taking place in the moment. As a result of competing interpretations about what the teacher enjoys, this was left blank, open to respondent’s interpretation. Focus on students. Dialogues about authenticity in relation to why the teacher went along with students in the discussion. Predominant interpretation by the teachers: you go along with them if they are really involved, following their interest, thinking for themselves. These elements included in the vignette.	I had presented students with an issue and subsequently I had the students respond to each other’s solutions. Because they were talking to each other, the class discussion went in a completely different direction than the one I had thought of. But I saw that they were following their interests and thinking about what was important to them [FS2]. Even though it deviated from the lesson objectives and I didn’t know what the conversation would lead to, I waited and followed them [AR1]. I enjoyed it so much [SA1] that I went along with their discussion completely.
#P3, Line 061: ‘And their conviction and the … black-and-white way of formulating. I was shocked sometimes, I think because I would express it much more nuanced myself. […] the opinions they expressed seemed so black and white, so thoughtless at first. Recently, students made rather racist remarks. I was annoyed, angry! But I also realized that I had to stay professional and tried to figure out [SE2]: what exactly do you think, how do you view this [FS3]? I didn't want to simply reprove them. […] I wanted to confront them with the fact that you can also view this differently, showing them different opinions [AR2]. I assume this was a conversation in which I made students think, and me too, I started to see new sides of them’.	Contextual aspects. Authentic situation according to the teachers, happens with political opinions. Self-awareness & Focus on students. Competing interpretation about whether getting angry is part of PiT. For that reason formulated more neutrally in the vignette (I was shocked). The core of PiT was considered the teacher correcting herself and consciously adopting an open attitude [SE2] to the perspectives of students [FS3]. Formulated this more explicitly in vignette. Acting responsively. It is unclear what the teacher does and therefore not adequate as an (authentic) representation of AR. Teachers’ interpretations are included to concretize the teacher’s actions in the vignette: stimulating students to listen to other viewpoints.	In this class discussion, a few students expressed opinions that shocked me. I didn't want to be guided by that feeling and instead react professionally. I consciously tried to have an open mind, pose questions and empathize with the students’ perspectives of the students [SE1]. That's how I learned to see the different sides the students have and discovered what’s important to them [FS3]. And at the same time, I realized that I wanted to show them that there is more to the world, by sharing multiple views. Of course, I don't know if I succeeded, but I did try to get everyone to listen to each other and to get different points of view into the discussion [AR2].

* All quotes and vignettes have been translated from Dutch.

The main challenges encountered concerned tensions between the insider and outsider perspectives, when teachers proposed changes to improve authenticity that were not consistent with the conceptualization of PiT. For example, a teacher argued that focus on students’ learning and their experiences happens simultaneously, which could not be taken into account given our structured design of the vignettes. Again, taking the insider perspectives of the teachers very seriously, the first author tried to reach agreement communicatively by mutually explaining, challenging and querying (implicit) understandings (Long 2017). Balancing the insider-outsider perspectives required extensive experimentation and dialogue about formulations with teachers and within the research team during meetings for researcher intersubjectivity.

When eventually teachers confirmed that the vignettes were clear and authentic representations of the conceptualization of PiT and only mentioned minor improvements on a detailed level, we decided that saturation had been achieved. In the end, one vignette of 75 −125 words with a narrative story-like structure was constructed for each combination.

Development of the survey structure and items

The aim was to measure the extent to which teachers experience PiT and recognize meanings of PiT in their daily practice, and to further explore variations and possible relations to respondent-specific characteristics. Accordingly, the survey was divided into three sections: (1) respondents’ experiences of PiT; (2) the meanings they attach to PiT and (3) respondent-specific characteristics. In the prefatory introduction, the respondents were provided with a short definition of PiT: ‘Teachers are fully attentive to what is happening in class, and actively respond to this’. The generation of survey items is explained below per section. All items were created by the first author and extensively discussed in meetings for researcher intersubjectivity.

Section 1 consisted of the 12 narrative vignettes that provided the operationalization of PiT. One Likert-type scale question was added to each vignette to elicit a judgment on respondents’ own similar experiences: ‘How often do you have an experience similar to this one?’ Descriptors for every scale point were included in order to provide clarity, to reduce ambiguity and to yield greater reliability (Brancato 2006).

Section 2 started with a reminder of the definition of PiT. The indicators (see Appendix B) were used to create two matrix questions, relating to the meanings of PiT for students and for the teachers themselves, respectively. Again, Likert-type scale items with descriptors were added. For both matrix questions an ‘other, please specify’ option was added. This is common in survey research into an emerging concept, when it is not certain whether the response categories are complete (Brancato 2006; see Figure 4 for an example of the matrix questions).

Figure 4. Example of matrix questions concerning the meanings of PiT.

In Section 3 we included the respondent-specific characteristics which data analysis in our earlier study (Roefs et al. 2021b) indicated as having a possible relationship with variations in teachers’ experiences of PiT. After all, we aimed to systematically investigate relationships between variations in teachers’ experiences of PiT and respondent-specific characteristics on a larger scale. Items related to gender, year of birth, and region were added. Table 3 presents the characteristics included in the survey.

Table 3. Respondent-specific characteristics included in the survey instrument.

Item	Respondent-specific characteristic	Question format	Options
1	Gender	Multiple choice	Woman; Man; Non-binary/third gender; Prefer not to respond
2	Year of birth	Numeric open-ended	N/A
3	Year in which started as a teacher	Numeric open-ended	N/A
4	Teaching subject	Check-all-that-apply, including also ‘other, please specify’	List of most common teaching subjects
5	Region of the school	Multiple choice	List of 12 regions in the Netherlands

Piloting, reviewing and revising the survey using narrative vignettes

We undertook three rounds of pilots with four teachers each, expert reviews and dialogues, and a final round of pilots with four teachers. The involvement of content experts was postponed to a final stage of the development process in order to ensure that revision and validation of the survey content would reside primarily in negotiation with teachers. The process continued until saturation was reached (see also Figure 1 for an overview of the process).

Initial three rounds of pilots

The objective of the pilots was to obtain qualitative information on how the vignettes and questions were interpreted and answered by actual respondents, in order to use this information as an entrance to a dialogue about possible sources of difficulties and a negotiation about improvements to the survey. Following Brancato’s (2006) procedure for cognitive interviews, the teachers completed a paper version of the survey, followed by a cognitive interview for which (probing) questions were prepared. These questions focused on the teachers’ experiences with the survey as a whole, to gain insight into the extent to which filling out the survey was found interesting and difficult. The teachers were also asked to think out loud retrospectively for each survey item and to articulate the thoughts and considerations that led to an answer. The subsequent dialogue depended on the outcomes of the cognitive interview. The participating teachers were selected via teacher educators and snowball sampling, aimed at including a variety of teachers.

In all rounds, filling out the survey was deemed interesting by most of the teachers involved, by stimulating reflection on their own teaching experiences. For some, the survey in particular provided language for what they already did and considered important. The cognitive interviews in the first round revealed that in Section 1, the teachers used their own PiT experiences as a frame of reference for answering the question relating to each vignette, as we intended, but only referred to one of the three constructs when justifying their choices. In Section 2, regarding the meanings of PiT, they no longer referred to experiences of PiT, but to general views on the quality of teaching. As a starting point for the dialogue, the first author identified these problems in relation to the research aims and conceptualization of PiT, and invited the teachers to share their thoughts about causes and solutions. Initially, the teachers were cautious and questioned their ability to contribute to the quality of the survey. During this process, the first author reflected within the research journal not only on substantive findings from the cognitive interviews, but also on the interactional process, searching for ways to empower the teachers. By addressing them as experts and emphasizing the significance of an open and critical dialogue for the co-construction of new understandings and solutions, an opening arose. The dialogues consisted of sharing and experimenting with conjectured alternatives, providing insights into causes and solutions for the survey’s design problems. For example, one teacher indicated that she could not determine one single answer based on the vignettes, because she recognized one construct more than the other. This resulted in the inclusion of a separate survey item for each construct (see Figure 5 for an example of a vignette and the corresponding three survey items). In a similar way, it became clear that the survey items concerning the meaning of PiT needed to be connected to a specific vignette as a frame of reference. These findings were input for discussions within the research team and final decisions on revisions, which were applied in the subsequent round of pilots. By trying to formulate new survey items sharply for Section 1, we discovered that the variants for Focus on students were not yet clearly distinguished, and adjusted the conceptualization (see the third framework in Figure 3).

Figure 5. Example of vignette and adapted corresponding survey items.

The pilots in the second round confirmed that the solutions worked as intended, and provided insights into the motivational aspects of the survey. Teachers gradually lost interest while filling out Section 1 (conceivably due to the increase in items per vignette). The research team decided to halve the number of vignettes in such a way that each variant occurred equally often in the vignette population, in as many varied combinations as possible. Table 4 shows the subpopulation of six vignettes selected according to the fractional factorial design we used for this purpose (Atzmüller and Steiner 2010).

Table 4. Application of the fractional factorial design used to halve the number of vignettes.

	Focus on students = FS1			Focus on students = FS2			Focus on students = FS3
	Acting resp. = AR1	Acting resp.  = AR2		Acting resp. = AR1	Acting resp. = AR2		Acting resp. = AR1	Acting resp. = AR2
Self-awareness = SE1		Vg2		Vg5				Vg4
Self-awareness = SE2	Vg1				Vg3		Vg6

‘Vg’ = vignette.

Within the third round, some items concerning the meaning of PiT – for example, the extent to which PiT is related to test and exam results – were found uninteresting and unpredictable. In collaboration with the teachers, the survey items regarding the meaning of PiT were specified in terms of what respondents may notice and experience in the moment of a PiT situation, which was considered more relevant.

Expert reviews and dialogues

The aim of the expert reviews and dialogues was content validation of the survey. The procedure was drawn from Grant and Davis’s (1997) approach to the selection and use of experts for instrument development. The experts were informed with a document defining and conceptualizing PiT, including the constructs and their variants. In addition, they received a review questionnaire to prevent confusion or bias. Using this questionnaire, the experts respectively assessed the representativeness and clarity of all vignettes and survey items and the comprehensiveness of the survey as a whole. ‘Measuring interrater agreement’ from the Grant and Davis (1997, 272) procedure was replaced with an extensive dialogue with the experts to build on their expertise as far as elaboration of possible improvements of the survey. Three experts were involved, who were familiar with (secondary) education and had expertise relating to the conceptual framework in several ways: research and publication on PiT (first expert); familiarity with PiT as teacher educator (second expert), and philosophical knowledge on presence and publications on related concepts (third expert).

According to the experts, the survey ‘gives a broad and articulate expression of PiT experiences’ (third expert), ‘showing that it is a holistic experience (of all constructs), and integrated with the instrumental side of teaching’ (first expert), and in which ‘a slippery concept has been made tangible’ (second expert). In the experts’ view, the vignettes offered respondents space for their own associations and the here-and-now formulations of the items would stimulate them to think about their own experiences and meanings. Thereby, the experts confirmed that the survey provided a comprehensive representation of the conceptualization of PiT and was closely related to educational practice, as we intended. The vignettes and items were largely positively rated for representativeness and clarity. Besides some minor issues, the main criticism pertained to the items relating to Acting responsively, the improvisational nature of which was insufficiently expressed. Discussing this issue with the experts, we realized that none of these items was related to the teacher’s particular attentiveness and awareness as described within the vignettes. The formulation of all survey items relating to Acting responsively was adapted accordingly (see for an example Figure 5, item 4.3).

Final round of pilots

As the survey seemed largely validated by the experts and teachers, at this stage, a web-based version of the survey (using Qualtrics) was designed and administered to four teachers. This round no longer yielded improvements to the content of the survey, but it did provide insights into the operation of the online survey. The layout was slightly adjusted for the sake of convenience. Three teachers explicitly mentioned that filling out the survey was valuable, because they felt stimulated to reflect on their experiences and beliefs in their own practice. The cognitive interviews showed that the teachers had considered all three constructs in determining answers for Section 1 and that they had related the meanings to particular PiT experiences within Section 2, both as we intended.

Subsequently, respondent validation of the entire survey was done by informing the teachers about the conceptualization of PiT and inviting them to be critical of the extent to which the vignettes and items represent this conceptualization and are related to educational practice. In the teachers’ view, the vignettes and items did represent the conceptualization of PiT and were realistic. Based on the findings of this final round, we decided that the survey was ready for administration.¹

Conclusions and discussion

This article discussed how the transition was made from qualitative data and findings to a large-scale survey instrument, to investigate experiences and meanings of PiT among a non-purposive sample of teachers. In order to ensure relevance for teachers, the survey instrument needed to reflect the richness of the qualitative data and findings, and relate closely to teachers’ daily practice. The start of the development process was the determination of the unique and varied characteristics involved in a fine-tuned conceptualization of PiT, drawn from qualitative findings and data from earlier studies (Roefs et al. 2021a, 2021b). A survey using narrative vignettes was developed to express these characteristics: the simultaneity of the constructs Self-awareness, Focus on students and Acting responsively, the variants of these constructs, and the intersubjective character of PiT. Qualitative data for the construction of the vignettes and items were systematically selected from the earlier studies according to a structured design (Evans et al. 2015).

Building on Onwuegbuzie et al. (2010), balancing insider-outsider perspectives was our first quality criterion, serving as an umbrella for the sub-criteria of respondent validation, expert validation and researcher intersubjectivity. We applied these methods using dialogues aimed at reaching understanding and agreement. Balancing insider-outsider perspectives contributed to the survey in three ways. First, by negotiating multiple ways of understanding the variations of PiT, predominantly with the 24 teachers involved, vignettes and survey items were constructed that on the one hand were specific enough to approximate the reality and nuances of the experience of PiT and, on the other hand, were abstract enough about classroom practices to allow respondents to form their own interpretations. Both are considered important features of vignettes (Poulou 2001). Second, by collaboratively (teachers and first author) generating a variety of alternatives as a means of problem-solving, surprising solutions were found for problems concerning the structure of the survey. These solutions improved the comprehensiveness of the survey as a whole. Third, the dialogues among the teachers and first author contributed to the motivational aspect of the survey, because teachers brought the perspective of what is important, relevant and interesting to teachers.

The intense collaboration with teachers posed challenges as well. As Torrance (2012, 120) suggested: ‘Respondent validation is no panacea, and it is in no sense a straightforward process. Quite the reverse, it is uncomfortable and challenging’. The challenges can mainly be traced back to an insider-outsider binary, in which the researcher felt a primary responsibility for connection to the conceptualization of PiT and the teachers primarily seemed to feel comfortable discussing own PiT experiences. Various attempts were made to move beyond this binary: addressing the teachers as experts, more disclosure of research information, and ‘opening a dialogic space’, as coined by Wegerif and Major (2019, 110). Researcher reflection, which was the second quality criteria we selected and used, was imperative here. The first author reflected not only on substantive findings from the development process, but also on the interactional process, searching for ways to see the teachers as worthy interlocutors.

Often, instruments are developed from an outsider perspective, in which respondents serve as informants for the researchers (cf. Howe 2004). Our approach to instrument development consisted of different types of interactions: between outsiders (researchers and experts) and the instrument; between insiders (teachers) and the instrument, and between outsider (first author) and insiders. This resulted in a process of instrument development that occurred in a dialectical relationship between the scientific concept and the everyday concept (in the intuitive understanding of teachers) of PiT (cf. Vygotsky 1987). This dialectical relationship seemed to contribute to capturing PiT within the survey in terms of the language and classroom practices of a variety of teachers. Furthermore, a continual focus on the situatedness of knowledge, following from our interpretivist stance, allowed for consistent decisions throughout the development process.

Nevertheless, transcending the insider-outsider binary could possibly be improved by critical reflection on and experimentation with the positionings and responsibilities of the researchers and participants within a collaborative process of instrument development. This raises several questions, which could be addressed in follow-up research. For example, at what stage of the development process should participants be involved? What would be more flexible positionings (e.g. insiders as researchers), building on Nakata's (2015) framework for insider-outsider positioning? And, how can the positionings and responsibilities of the researchers and participants be defined? This could include refinement of the quality criteria.

Implications for educational researchers

With this article, we aimed to contribute to methodological knowledge on the application of mixed methods in quantitative instrument development. What we have added is insight into the transition from qualitative findings and data to the development of an instrument; a process that is often carried out, but rarely explicitly described (cf. Alise and Teddlie 2010; Onwuegbuzie et al. 2010; Maxwell 2016). The criteria used, (justifications for) the steps taken, the challenges encountered and solutions found may be informative for educational researchers in several ways.

First, the insights into the transition process may contribute to a mindful approach to instrument development. In particular, ‘working with the categories or constructs one knows and simultaneously being open to new meanings’, as mentioned by Dellinger and Leech (2007, 320), is a crucial quality of the mixed-methods researcher. Second, these insights may be relevant for educational researchers, who aim to investigate emerging concepts – for which little theory and no reference instruments are available – on a large scale. This also applies to educational researchers who work from an interpretivist worldview and believe that for intrasubjective and intersubjective experiences the aim is not that ‘a single ‘true’ (‘valid’) account can (and should) be produced’ (Torrance 2012, 114). These researchers may limit themselves to qualitative research. Our discussions and reflections may open up possibilities for those engaged in large-scale research, for example, aimed at gaining insight in the broader resonance and relevance of an emerging concept. Third, the collaboration of researchers with teachers and experts within the development process allowed them to develop a shared understanding of PiT and to capture PiT within the survey using the language and classroom practices of teachers. Educational researchers could harness the value of this collaboration across all stages of the research process by using co-design approaches. Similar to our development process, co-design builds on the complementary strengths of researchers and practitioners or end users, connecting abstract ideas and principles with concrete experiences or instances (Zamenopoulos and Alexiou 2018; Trischler et al. 2019; Slattery et al. 2020). Insights into research collaboration and solutions for the ethical dilemmas involved (Goodyear-Smith et al. 2015) may therefore contribute to reducing the divide between research and practice and making research findings relevant and useful for both researchers and practitioners.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by The Dutch Research Council (NWO) under grant number 023.007.028.

Notes

1 If you would like to obtain a copy of the survey, please contact the corresponding author.