Synthesisability and optimising exploratory action research for continuing professional development

ABSTRACT

Teachers need to engage in effective professional learning, and Exploratory Action Research (EAR) supports Continuing Professional Development (CPD) in building teachers’ knowledge and skills, even in low-resourced environments. This paper first explores mentored EAR for CPD, its affordances, and challenges. Asking EAR to serve too many functions can frustrate efforts if teachers view research as taxing, or academics undervalue teachers’ outputs. One way of optimising CPD is to shift the focus onto particular concepts of research quality to help teachers and academics develop professionally. This paper next conceptually examines the conjunction of EAR with Qualitative Research Synthesis (QRS), a secondary research method that synthesises and offers new perspectives on aggregated qualitative research outputs. A critical exploration of a published study applying QRS methods in a related field, Technology-Mediated Task Based Language Teaching, is given. The analysis suggests that a novel research orientation be adopted- that of ‘synthesisability’. The contribution to the field is amelioration of the researcher-practitioner relationship, and the pedagogical implications are that teachers can show the thoroughness and value of their impactful work.

KEYWORDS:

• Exploratory action research
• synthesisability
• qualitative research synthesis
• language teacher professional development

Introduction

Exploratory Action Research (EAR) is an established method for teachers to take charge of their own professional learning through Continuing Professional Development (CPD), working with authority to decide research problems and evaluating context-specific solutions. As shown in an Educational Action Research journal special issue (Consoli and Dikilitaş 2021), there are examples of exploratory practitioner research which work especially well in promoting the professional learning of language teachers. However, there are few examples of such learning at scale that impact practice beyond the original context. An exception is an attempt from an EAR mentoring scheme in Chile that had local and global impacts. Going beyond the learning of individual teachers, the British Council selected third-person accounts of teachers’ research after participating teachers presented their research to one another in the Champion Teachers Project (CTP) (Rebolledo, Smith, and Bullock 2016). This shows the potential of EAR to scale and transfer to other contexts: to date, 200 Chilean teachers completed the project, affecting 6000 students, and it has expanded to other low-resourced contexts (R. Smith and Rebolledo 2022).

Part of the appeal of EAR for CPD is in its initial exploratory phase, preceding optional cycles of action research (AR) (R. Smith and Rebolledo 2022). In this phase investigators reflect broadly upon what new learning they want to derive, generate tentative initial questions, and finally reflect upon their observations. This functions as a puzzling (Hanks 2017) tool, informing the prospective investigator’s decision about whether to proceed with the AR phase. In the CTP example, these decisions were scaffolded by mentors. This use of EAR combines the reflective systematicity of AR with more exploratory flexibility. Openness to new directions may make EAR suitable for CPD, while still enjoying the benefits of AR more generally such as systematic inquiry, being conducted by professionals in their context, and engendering locally meaningful change by way of proceeding cyclically towards a defined investigatory target to which responses are planned, acted upon, and results observed, informing a new stage of reflection beckoning the next investigative cycle.

However, while it is impactful, the CTP example required a dedicated research team using the full resources of the British Council. Furthermore, state-of-the-art academic or professional literatures may be inaccessibly paywalled, or teachers may lack the luxuries of time or support to engage with them. Alternatively, the growing popularity of Qualitative Research Synthesis (QRS) shows potential for achieving similar benefits of scalability to EAR, but which can be conducted by less well-resourced practitioners and synthesists. Recently, in language learning and teaching QRS has been suggested for summarising qualitative findings like perceptions commonly generated by small-scale action or exploratory research (Chong and Plonsky 2021). This secondary research method has also been used in healthcare to synthesise and offer novel insights into stakeholder experiences (Barradell and Bell 2021). A systematic literature search is conducted, and qualitative findings are extracted from studies meeting the inclusion criteria before being synthesised and reported. The emancipatory potential of QRS to summarise and synthesise while simultaneously promoting locally meaningful knowledge and skills in the context of professional learning is worthy of further investigation.

One of the key challenges for using QRS with EAR is what I term synthesisability: conducting an EAR project so as to make your work more synthesisable. Efforts are already being made to guide synthesists in how to optimise EAR studies when setting inclusion criteria or evaluating study quality (Chong and Plonsky 2021), but this paper considers the opposite angle: how EAR authors can make their rigour and contribution more apparent to give their work the best chance of having an impact through being included in QRS studies. To achieve this, this paper demonstrates how a detailed understanding of these two research methods, EAR and QRS, can be mutually beneficial for both synthesists and practitioners. This paper uses a prominent example of a QRS from the field of language teaching to explore how synthesisability can be of value to these different stakeholders. Initially however, EAR for CPD must be examined in detail.

Exploratory action research for continuing professional development

Hanks (2017) includes EAR in the ‘family tree’ of practitioner research alongside narrative enquiry and teacher research but separate from AR and Exploratory Practice (EP). EP promotes exploratory investigations for understanding rather than solutions (Allwright 2003, 2005; Allwright and Hanks 2009; Hanks 2017). EAR amalgamates AR and EP (R. Smith 2015) – prefacing an AR cycle with exploratory, understanding-oriented investigations. As such, EAR resembles AR and EP as a hybrid form of practitioner research for professional learning that prioritises collegiality, understanding, and praxis while optimising the advantages of its component methods by their sequential ordering and flexible application, the AR cycle being optional. Smith’s experience-based assumption was that teachers would generate personal theories from systematic reflective-practice.

The second embedded assumption was that personal experience alters teachers’ views of CPD and research. Previous encounters with traditional CPD may have fostered negative attitudes, making engagement with even simplified summaries like OASIS (Marsden et al. 2018) challenging. EAR for CPD aims for evidence-led practice to represent individualised pathways to overcoming attitudinal barriers to continued ‘engagement in and with’ research (Borg 2010, 392), while also providing the access, time, and need to engage. This may have lasting impacts. This paper will now explore the affordances, challenges, and optimisation of this method.

Affordances

Cohen, Manion, and Morrison (2018) synthesised procedural frameworks from AR literature. Their analysis suggested that AR follow an eight-stage process grounded in constant reflection: problem identification, stakeholder discussions, literature reference, revising understanding, selecting procedures, deciding evaluation criteria, implementation, and success assessment. Mentored EAR for CPD may scaffolded teachers’ progression through these steps. By doing so, practitioner research can play crucial technical (efficiency), practical (appropriate action), and emancipatory (social awareness-building) functions for teachers (Burns 2005; Carr and Kemmis 2003). As with AR, it can be grounded in teachers’ own concerns (Ferrance 2000).

One element that differentiates EAR for CPD is the early exploratory phase. Smith and Rebolledo (2022) claim teachers take more appropriate, less disruptive actions, and gain pre- and posttest perspectives on their teaching. While clarity about mentor’s roles is needed, methodological trial and error in the context of ongoing, dialogic, reflective collaboration could be crucial for teachers’ decision-making, as in Bustos-Moraga and Mann (2022). However, it is debatable whether EAR for CPD could be described as non-disruptive. The high dropout rate in the CTP example is one measure of teachers’ perceptions of the effort required (R. Smith, Connelly, and Rebolledo 2014). Nevertheless, teachers’ reflections in and on action (Schön 1983) are scaffolded by the mentoring architecture to develop habitual reflection alongside practice (Arnold 2020). Ultimately the practical benefits of this may be worthwhile. Smith and Rebolledo (2022) summarised unpublished 2017 analyses of self-report data from teachers in CTP cohorts 1–4, reporting positive impacts on teaching, empathy, research skills, and classroom participation. Mentored EAR for CPD may enable teachers to identify improvable areas within their teaching that promote synthesisability and enhance impact.

Challenges

Competing tensions in mentored EAR for CPD relate to agency. The assumption is that novice practitioner-researchers would be overwhelmed without guidance (Dikilitaş and Wyatt 2018). Smith et al. (2014) describes the mentoring architecture used in the CTP example. Teachers work through research over one year. However, the mentor’s decision-making influence is unclear. Whereas practitioner research can suffer from academic expectations (R. Smith 2015), mentored teacher research aims to bridge the researcher-practitioner divide by providing direct psychological and technical support- making research more accessible (Dikilitaş and Wyatt 2018). However, teachers have preconceived notions of research as formal, detached, scientific inquiry (Borg 2009), contrasted with the key AR concern of contextual relevance and alignment with teachers’ practical experience (Burns 2005). Practitioner-researchers negotiate similar academic expectations (Slimani-Rolls and Kiely 2019; R. Smith, Connelly, and Rebolledo 2014). Mentors may also have preconceived notions that research should be academic, requiring engagement with the literature which EAR does not formally promote (R. Smith and Rebolledo 2022). Although a strong professional knowledge base is part of what makes teaching a profession, before the CTP many teachers simply lacked the time, or access to CPD which promoted ongoing interest or engagement with it (R. Smith, Connelly, and Rebolledo 2014). However, EAR for CPD aims to foster teachers’ research mindsets towards their work. In the CTP, after first identifying issues in their context teachers can choose whether to engage with literature without necessitating it – a delicate balancing act. Voluntary engagement with the literature could be mediated through the mentor. However, if interaction with scholarship is premature or imposed, it pre-empts teachers’ voluntary engagement. Guidance for support without disenfranchising teachers is needed (R. Smith 2020).

Ethical considerations may be underemphasised or relativised in school-based research (Bryan and Burstow 2018). Nevertheless, AR practitioners must address similar issues as academic researchers- consent, confidentiality, and autonomy. Hanks (2017) argues that tensions between anonymity and emancipation and other issues are often underattended-to compared with theoretical issues or research-practice boundary delineation, and Davison, Martinsons, and Wong (2022) concur. In EAR for CPD, certain activities may be part of teachers’ Zone of Accepted Practice (ZAP) (Zeni 1998). That is, undertaking and reporting reasonable quality-of-service evaluations as accepted parts of teaching. Their ultimate purpose is learners’ educational benefit, and research benefits that accrue are subsidiary. Nevertheless, most studies do not directly contend with ethics (Rebolledo, Smith, and Bullock 2016). It is not stated what teachers do as teachers and as researchers. Issues surrounding voluntary informed consent with vulnerable minors, dependent teacher-student relationships and participation pressure, or consultations with competent guardians go unmentioned. Stakeholder involvement was part of the AR framework proposed in Cohen, Manion, and Morrison (2018). This area of EAR for CPD must be more developed.

Optimisation

EAR for CPD faces challenges. Teachers may see engagement in research as burdensome without incentives (R. Smith et al. 2015) or institutional scaffolding (Slimani-Rolls and Kiely 2019). That said, such projects are crucial for promoting systematic professional learning – impacting practice. ‘Teachers need to learn to think about their teaching and examine what they are doing in ways that may be different from what they were used to’ (Paran 2017, 507). One additional benefit could be teachers’ findings or research decisions. These may be overlooked by academics who think teachers cannot theorise, be critical, or systematic as researchers (Burns 2005; Hanks 2022). Such studies are often small, classroom-based, qualitative, contextualised, and of variable quality. Of course, most EAR projects do not aim at generalisable results. However, at scale, and under appropriate conditions, such findings could represent valuable untapped information. Cohen, Manion, and Morrison (2018) and others have proposed standards for rigorous AR, however these are difficult to use unaided. Simply evaluating projects subjectively is equally unhelpful. Alternatively, generating robust impact evidence beyond published booklets of practitioner research, such as Fully Inclusive Practitioner Research (2024), may lend additional credibility to initiatives which leverage academic and institutional power to enfranchise teachers through CPD, particularly in low-resourced environments. Such projects may have prestige attached. Professional advancement may also be a motivating factor in teachers’ engagement with research (Medgyes 2017). The same is true for researchers. Projects that both constitute good EAR for CPD, facilitate engagement in and with research (Borg 2010), and also generate robust results might incentivise participation if linked to career development.

In recent years calls have grown to bridge gulfs between teaching practitioners and academics to optimise research impacts (Sato and Loewen 2022). Adopting innovative research methodologies could provide support by summarising research findings (Chong 2020). Additionally, practitioners could inform research agendas (Leow et al. 2022), realigning notions of impact (Hanks 2022), and placing practitioners at the heart of knowledge generation – a key goal of (CARN 2023). One field which resembles EAR for CPD and has already benefited from such an approach is Technology-Mediated Task Based Language Teaching (TMTBLT). Interest in this blend of Task Based Language Teaching (TBLT) and Computer Assisted Language Learning (CALL) flourishes alongside technological expansion. Benefits include increased positive affect (Reinders and Wattana 2015), motivation (Tavakoli, Reza Lotfi, and Biria 2019), outcomes (Solares 2014), and noticing feedback (Gurzynski-Weiss and Baralt 2015). However, the research agenda needed updating.

One pertinent example, Chong and Reinders (2020) (hereafter, ‘the study’) applies a novel, synthetic approach to secondary research. Its justification was that TMTBLT research was often not replicable. Following calls for greater systematisation (Ziegler 2016), and previous meta-analyses (Lara and McKay 2019), the study summarises qualitative primary research. Its questions assess TMTBLT studies’ characteristics, aiming to identify emergent themes. Using a QRS method, it aggregates data from 16 studies. Inductive coding produces a summative Grounded Theory (GT) (Glaser and Strauss 1967), five affordances, and three limitations. Of course, GT generates thick description, whereas QRS filters and selects study highlights, leading to some conceptual tension. However, the study finds that if contextual factors allow, TMTBLT can help learners of English to interact, and build language and other skills (Chong and Reinders 2020). A future research agenda is then proposed.

There are methodological similarities and issues faced by EAR and TMTBLT. Could synthesisability be a useful conceptual orientation for EAR? The potential of integrating QRS methods into EAR for CPD has not yet been addressed. Accordingly, the next section of this paper gives a detailed methodological critique of Chong and Reinders’ (2020) study. The study and its impact are contextualised. The suitability of its methods and questions are then evaluated. Clear research and practice implications are given. Following this, the final part of this paper draws conceptual lessons for EAR. The suitability of QRS as a method based on synthesisability as a conceptual orientation is addressed. Finally, recommendations for designing synthesisable practitioner research are given.

Critique of a published QRS study

Relations to the literature

It has been claimed that ‘The influence of technology on … TBLT in particular cannot be overstated’ (Jackson 2022, 17). The term TMTBLT was coined by González-Lloret and Ortega (2014), and refers to meaning-focused language activities that are facilitated by technology, communicative, goal-oriented, learner-driven, authentic, and include reflective elements. TMTBLT differentiates pure language tasks from those which build digital skills (González-Lloret 2015). Curricula based on TMTBLT incorporate technology as a full, rather than adjacent, element (B. Smith and González-Lloret 2021). Considering the ubiquity of text, audio, and video technologies, their impact on language education needs examination (González-Lloret and Ziegler 2021). Describing research needs, Révész (2021) suggests that quasi-experimental, descriptive, case study, and AR designs are commonly adopted.

Originality

In a paper on TMTBLT, Smith and González-Lloret (2021, 531–532) state, ‘it is essential to be able to aggregate findings into more robust results that allow us to state with confidence how tasks and technologies work’. Indeed, systematic reviews and meta-analyses are considered by some authors the most reliable form of research evidence (John and McNeal 2017). Révész (2021) concurred that systematic research synthesis is key in establishing TMTBLT. In TBLT, meta-analysis of 52 studies demonstrated positive impacts on learning outcomes (Lara and McKay 2019). In CALL, meta-analysis of 64 studies demonstrated enhanced writing performance (Seyyedrezaei et al. 2022). Furthermore, umbrella review of 16 QRS studies found it an important emerging research method in CALL (Chong and Reinders 2021). In this context, Chong and Reinders (2020) QRS study appears well-timed.

QRS, also sometimes referred to as meta-synthesis (Siddaway, Wood, and Hedges 2019) or qualitative evidence synthesis (Lewin et al. 2018), amalgamates results from qualitative or mixed-method case studies (Chong and Plonsky 2021). While meta-synthesis has a long history in nursing (Walsh and Downe 2005), the study represents an original methodologic contribution to TMTBLT. Its GT and list of affordances and limitations stated that TMTBLT fostered collaboration, positive attitudes, student-centred learning and both language and non-language skills. Conversely, concerns were raised about difficulty, time, and workload. These themes were narrowly identifiable in each paper. In Chen and Lynn Brown (2012), interviews revealed that six ESL learners viewed computer-mediated writing activities positively. Chong and Reinders (2020) contended that while small sample size complicated generalisability in some individual papers, their QRS highlighted positive impacts on affect in several papers. The authors claimed broader lessons could thus be drawn, despite the study comprising only 16 papers. Nevertheless, these results concurred with previous research on positive affect in TMTBLT (Reinders and Wattana 2015), acting as, ‘transferability … a QUAL analogue to external validity’ (Tashakkori, Johnson, and Teddlie 2020, 34).

Criticism

As a recent addition to the literature, there are no published criticisms of the study. However, there are long-standing philosophical criticisms from the field of nursing, where qualitative meta-synthesis has a history. Researchers rooted in relativistic ontological, or subjectivist epistemological traditions (Moon and Blackman 2014) fundamentally disagree with synthetic qualitative research. Qualitative research gives rich description of smaller, naturally occurring settings (Mackey and Gass 2015). Researchers who dispute QRS’s trustworthiness argue that while aggregation works in positivist research, the context-specific nature of qualitative findings is lost in synthesis (Sandelowski 2006; Sandelowski, Docherty, and Emden 1997). In effect, it turns ‘rich descriptions into thin abstractions that are of little use to nursing practice knowledge and does not advance nursing science’ (Bergdahl 2019, 7). For a detailed overview of philosophical objections, see Elizabeth et al. (2016).

Impact

The journal, Language Learning and Technology, is a peer-reviewed, open-access journal whose audience is language educators (Language Learning and Technology, n.d.). Peer-review ensures that research claims are supported by concrete analysis (Garcia-Costa et al. 2022) and open-access articles are viewed, downloaded, and cited more than paywalled ones (Draux, Lucraft, and Walker 2018). This journal’s Impact Factor has risen from 2.113 in 2017 to 4.694 in 2021, placing it 10^th out of 194 linguistics journals and 31^st out of 267 education journals (Language Learning and Technology, n.d.). As research which aims to inform practitioners, the journal choice is appropriate and enhanced impact.

A citation search was conducted on the twenty-third of November 2022, on Scopus. The study was cited five times (2021) and 18 times (2022) for a total of 23 citations. Those values fell to three (2021) and 12 (2022) for a total of 15 citations when self-citations were excluded. The rapid increases in citations could be considered a measure of impact. Of the 15, 12 were from refereed journal articles. Seven were in open-access journals. Three were currently in press. This is an additional measure of the study’s ongoing impact.

Another, more interpretative, conception could be evaluating substantiveness. Does a citation leverage the results or methodology of the study to great effect? Gerashchenko et al. (2021, 170) write, ‘As Chong and Reinders state it, “computer-assisted language learning (CALL) has grown exponentially as a field, with an increasing number of studies, mostly focusing on second/foreign language”’. This citation could be considered non-substantive, as it doesn’t leverage the power of the GT, affordances and limitations, or the method. Conversely, Aubrey and Philpott (2022, 6) write, ‘Chong and Reinders’ (2020) review of research identified several benefits, which include the enhancement of collaboration, more positive attitudes towards language learning, the cultivation of student-centred learning, and the development of non-language skills (e.g. technology and intercultural literacy).’ This citation is more substantive, directly using the study findings.

Suitability

The study’s research questions are ‘1. What are the characteristics of technology-mediated tasks in the primary studies? 2. What are the affordances and limitations of technology-mediated tasks reported in these studies? 3. What are other emergent themes resulting from the GT analysis?’ (Chong and Reinders 2020, 71). These carefully bounded research questions are answerable through the study’s interpretivist and explanatory analysis. Rather than starting with a hypothesis, the researchers induct one from available evidence (T. Chen 2016). These are the terms of the emergent GT. Flemming and Noyes (2021, 2) write that QRS is well suited to answering, ‘questions that seek to enhance understanding of a particular phenomenon of interest’. The study’s well-aligned methodology and questions enhance the trustworthiness of its findings by integrating themes from 16 papers. Lastly, unlike a systematic review no critical analysis is given. However, QRS involves methodical and exhaustive searches, clear inclusion and synthesis procedures, and reliability enhancement through prior agreement on benchmarking. In this case, the authors used the Qualitative Research Guidelines of Journals of Language Learning and Technology and TESOL Quarterly (Chong and Plonsky 2021). Accordingly, the paradigm’s methodological rigour makes it appropriate for the considered questions.

Again, those with philosophical objections to meta-synthetic qualitative analysis may disagree, and other limitations exist like English bias. Only two papers do not focus on English language teaching (Chong and Reinders 2020). Finally, publication bias cannot be ignored. Unlike Chen (2016), the study cites only published works, excluding formats like conference papers where important practitioner research may be presented. Nevertheless, the careful matching of paradigm, methodology, questions, and systematic reviews being considered by some to be atop the evidentiary pyramid (John and McNeal 2017), may give confidence in the replicability of the findings.

Ethics

When conducting qualitative secondary research, researchers must fulfil duties of care to the original data sources, despite being removed from that context (Irwin 2013). Ethical standards are typically established by academic institutions or peer-reviewed publications. No ethical issues, such as deanonymizing research participants, have been flagged with the study. However, the status of QRS as a filtering method may have its own ethical implications. The study was published open access. Open access research has clear benefits for the field. However, the studies included in a QRS may be paywalled, and less accessible to practitioners without institutional access. In effect, as the creator of a readily accessible summary, the synthesist will curate the official record on TMTBLT. QRS authors have an ethical duty to interpret studies judiciously.

Transparency

The study carefully explains the seven-stage process undertaken to locate, sort, select, and analyse papers. The authors follow recommendations from Chen (2016). A side-by-side comparison of similarities and differences is provided in Table 1. In some respects, Chong and Reinders’ (2020) study is explained more transparently than Chen (2016). For example, the use of NVivo software for data extraction and synthesis (Chong and Plonsky 2021) is stated. Equivalent procedures go undescribed in the earlier paper. Conversely, in Chen (2016) the time spent coding, number and format of coder meetings, and extent of collaboration is described in detail. This choice may reflect epistemological disparities. Chong and Reinders (2020) opted to deemphasise subjective and practical aspects of the coding experience which may have subtly impacted their induction of codes and themes. Indeed, in a later scoping review by the same authors (Chong and Reinders 2022), discussions about disagreement resolution procedures for exclusion decisions were more clearly described- they occurred via emails and WhatsApp. The authors likely realised that readers of their 2020 study could have benefited from clarity on how author interactions were negotiated, as it impacts interpretation, and amended this practice. Nonetheless, the procedures in the study are described clearly, and align with the recommendations of Chong and Plonsky (2021). For example, codes and themes are provided as supplementary materials, and coding and analysis stages are represented diagrammatically. Overall, the portrayal of procedures enhances transparency by being clear and replicable.

Table 1. A comparison of the qualitative synthesis methods adopted by two studies.

	Chen (2016)	Chong and Reinders (2020)
Identify relevant sources	Review articles Studies Databases Journals Conference programmes and proceedings World Wide Web	Digital libraries and databases Major refereed journals World Wide Web Primary studies in book chapters
Identify keywords and search	Search string and source titles given in paper
Evaluation	Publication retrieval rate, screening procedures, exclusion rate and criteria justified against Qualitative Research Guidelines of Journals of Language Learning and Technology and TESOL Quarterly
Extraction tools	Not given	NVivo
Research synthesis: coding	Two coders closely collaborate. Agreement reached on procedures. Substantive face-to-face collaborative coding on five studies. Remaining 15 studies coded independently. Last, theoretical (thematic) coding occurred.	One coder identifies codes in 16 studies. Next, conceptual and descriptive themes and sub themes identified. Last, second author gave input.
Analysis and reporting	Memo writing, memo and code sorting, and theory writing.	Initial, focused, and axial coding. Memo writing and constant comparison.

Evaluative framework

There are over 30 methods of conducting QRS (Flemming and Noyes 2021). Various best-practice frameworks propose to guide quality decision-making, including GRADE CERQual (Lewin et al. 2018) – Confidence in the Evidence from Reviews of Qualitative research. As for reporting, with reference to the current study Chong and Plonsky (2021, 1029), state, ‘reviewers are recommended to adopt … Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA).’ Accordingly, this critique judges the study on these terms. PRISMA is an evidence-based reporting standard, whose application encourages sound systematic reviewing (Page et al. 2021), and spans 27 items over seven areas in typical manuscripts- Titles, Methods, Discussion, etc.

One checklist section covers 12 items and six areas required for abstracts. ‘A structured summary covering the rationale and objective’ alongside data sources, extraction methods, results, and conclusion should be given, creating an independent summary (Athikarisamy and Patole 2021, 114). A point-by-point analysis of the study’s abstract against the checklist is informative. The first line clearly identifies the study as a research synthesis. Next, the abstract gives a rationale, ‘naturalistic, classroom-based studies … whose findings may be less generalizable … a qualitative research synthesis … is warranted’, research objective, ‘advancing the development of the field and informing practice’, and sources, ‘qualitative findings from 16 technology-mediated TBLT studies published between 2002 and 2017 in second and foreign language contexts’ (Chong and Reinders 2020, 70). It also identifies the synthesis method as GT. Finally, an interpretive account of results and implications is given. Unfortunately, other PRISMA abstract checklist elements are absent. Study selection criteria, search dates, methods of data extraction, bias risk assessment, summaries of evidence limitations, and funding sources are unspecified. However, meeting around half the criteria suggests a rigorous social science application of PRISMA, and to generally high study quality. Omissions may result from journal word count or scope limitations. Furthermore, PRIMSA aims to improve reporting in medicine- a field with positivist traditions and entrenched reporting conventions. Thus, differences with educational reporting norms may also explain discrepancies. While borrowing PRISMA may speak to the prestige of the natural sciences, bespoke frameworks might better suit QRS for practitioner research in TMTBLT or EAR.

Recommendations

This critique highlighted strengths and limitations of a QRS study of TMTBLT, Chong and Reinders (2020). Despite the short time since publication, the study has had unprecedented impacts on evidencing TMTBLT, promoting replicable research. Complementing established aggregative quantitative methodologies, the novel approach aims to bridge gulfs between researchers and practitioners. The method might isolate future research directions by identifying previously uninvestigated topics. Nevertheless, areas for improvement remain. Those grounded in philosophical objections may be unresolvable. However, in other areas concrete recommendations for addressing limitations and further developing strengths can now be discussed with reference to current literature.

Research implications

This critique has demonstrated that the major strength of QRS is addressing calls for greater systematisation (Ziegler 2016) and aggregation (B. Smith and González-Lloret 2021) to explicate the benefits of TMTBLT. The nature of language education lends itself to case study-based research. This is unlikely to change. However, one practical recommendation is that multiple TMTBLT QRS be performed. Each only looks at 15 to 20 papers, and many different forms exist (Flemming and Noyes 2021). Doing so would further increase trustworthiness, ameliorating the tension in making generalisability claims based on a small sample of 16 studies.

The study also has limitations. One issue is the ‘file drawer problem’ (Rosenthal 1979, 638 as cited in T. Chen 2016) – publication bias. The study aims to evidence a field where many studies remain small and highly contextualised. It is possible that much innovative, yet unpublished, work is being done by TMTBLT educators that is simply not being reported. As Table 1 shows, this approach was previously adopted in Chen (2016). Therefore, a practical recommendation for future QRS would be to consider publications like post-conference proceedings, Special Interest Group newsletters, or indeed practitioner research resulting from EAR for CPD projects.

Another limitation may be that other qualitative synthesis formats are more effective at communicating research. The dedication of a special issue of The Modern Language Journal (Sato and Loewen 2022) to improving the researcher-practitioner relationship shows its importance. Indeed, one of the study authors has written separately about the topic’s importance (Chong 2020). Looking again at the study and others (Chong and Reinders 2021), jargon-heavy descriptions of database searching, coding procedures, and synthesis methods abound. A busy teacher may not skim-read a QRS, no matter how transparent the procedural explanation. Révész (2021), acknowledging the importance of research synthesis for TMTBLT, states that narrative reviews may be more comprehensible for teachers. Therefore, a concrete recommendation is for researchers to further synthesise their QRS research findings into alternative, more palatable forms for busy teachers engaged in EAR for CPD. However, as with the synthesist-as-gatekeeper, positioning a few individuals as summarizers and curators of the literature does carry its own inherent risks.

Summary

A critique of Chong and Reinders (2020) highlighted the original and ongoing contribution made by the study since publication. Analysis of its relations to existing scholarship, methodological appropriateness, and application of research methods indicated that the study was well-timed, well-conceived, rigorously designed, and carefully executed. The study has had, and continues to have, a much-needed legitimating impact on research evidence in the field of TMTBLT. Lastly, discussion of the study’s research implications revealed practical directions of future study which build on its strengths and address its limitations. QRS has revealed its potential as an additional plank in the bridge spanning the gap between researchers and teaching practitioners in this field of language education. Now the potential affordances of practitioners, researchers, and practitioner-researchers leveraging the concept of synthesisability and QRS methods for EAR will be addressed.

Synthesisability

A conceptual orientation for EAR

This paper first described how EAR promotes teachers’ CPD through initiatives like the CTP. Following this, paper demonstrated how QRS can be applied to TMTBLT. TMTBLT research resembles EAR for CPD. Small-scale case studies, producing mainly qualitative findings are common. However, as with its application in the field of healthcare, QRS has produced generalisable evidence to stand alongside localised research. For CPD-focused practitioner research, this paper now proposes a new research orientation- synthesisability. A speculative model is shown in Figure 1.

Figure 1. A conceptual diagram representing proposed characteristics of an alternative research orientation- synthesisability.

Synthesisability overlaps EAR for CPD in some areas. Synthesisable studies are practice focused. Intersecting domains of research and practice produce salient results for everyone. Synthesisable studies are doubly knowledge-generative, that is, producing reflective phenomenological information and empirical information. The study location is crucial for this. Another similar area would be methodological. Synthesisable studies produce qualitative or mixed-method findings – a prerequisite for QRS, and likely achievable for teachers. An area of difference might be time. Synthesisable studies could be iterative like EAR, but not exclusively so, although research processes often are developmental. For EAR alone the rationale for this approach is clear. Synthesisable studies are deeply co-operative. Practitioner-researchers, embedded practitioners and researchers, or teams of practitioners, learners and researchers could work closely. This extends to deciding the outputs generated. Giving proportionate agency to control the narrative occurs through co-operation and communication. First (individual), second (co-operative), and third-person (broader impact) modes of AR inquiry (Hynes 2012) are thus employed- making synthesisability a good fit for EAR for CPD.

However, one feature of outsized importance (See Figure 1) is standardisation – in other words, quality control. For example, there must be inclusion criteria. Synthesisable primary studies must be designed against these criteria, the formal details of which could be negotiated project-by-project. For example, some EAR is ethically underdeveloped. In the CTP, the researchers positioned themselves as gatekeepers of teachers’ stories, ghost-writing the final third-person accounts of the project’s research outputs from the teachers’ notes for online publication. QRS is similarly vulnerable. The synthesist is the curator of the official record on the subject. By mandating (and clearly stating) that all stakeholders were involved at every stage, synthesisable studies can be more ethically sound, and can marry rigour with individual discovery.

Operationalising synthesisability in EAR for CPD

Operationalising synthesisability requires considerable researcher-practitioner collaboration. Every stage will be systematized and guided. This played a similar role to establishing inclusion criteria before synthesis in QRS. Standardisation achieved through co-operation is also crucial (See Figure 1). One way to operationalise this is output. Booklets of practitioner research can be produced to provide encouragement and ideas to future teachers either directly or via mentors. Their filtering function is determined by the intended audience- as with a project like OASIS (Marsden et al. 2018). It could communicate research findings to lay audiences. Conversely, a QRS is really written for a research audience. Recommendations for synthesists are offered in Table 2. Producing multiple outputs for different audiences from the same set of projects could ameliorate the conceptual tension between the local and personal narratives generated by teachers’ EAR on the one hand, and the aggregative aims of a QRS on the other. Rather than merely summarising what has been done, the QRS can derive new meaning from the synthesised work (Barradell and Bell 2021). Methodical examination offers new perspectives from and on teachers’ research, while still respecting its value as standalone EAR. To the best of this author’s knowledge, even in the fields of healthcare and TMTBLT where QRS has been applied, no project has attempted to produce dual outputs by operationalising synthesisability as part of a harmonised endeavour. Doing so would generate resources that act as both practitioner primer for EAR and as a complement to future academic research efforts. Additionally, generating QRS work of an academically publishable standard may incentivise academics to support practitioner researcher projects, as in the example of the CTP. In low-resourced environments, this may lend additional valuable institutional credibility, as well as mentoring support, to the project. Lastly, by having practitioners lead the EAR which in turn informs the researcher’s QRS those teachers are, in effect, setting the research agenda, bridging the research-practice gap (Sato and Loewen 2022), and further validating teachers’ research efforts.

Table 2. A table describing recommendations for operationalisation of synthesisable EAR for CPD.

Recommendation	Details
(1) Establish CPD initiatives	Leverage privileged status to obtain institutional support.
(2) Promote teachers’ choice	Ask teachers to propose research agendas of interest to them. Group topics thematically for synthesis across cohorts if necessary.
(3) Standardise support	Provide high levels of ongoing, standardised support. Assisting with study development and ethical matters like stakeholder collaboration will promote synthesisability through quality control.
(4) Clarify submission formats	Teachers submit standardised accounts.
(5) Diversify output formats	Produce outputs for different audiences. For example, a similar booklet to Champion Teachers: stories … for teachers, and a QRS for both research and teacher audiences.

Viewed through a synthesisability lens, recommendations can be made for prospective practitioner-researchers engaging in EAR for CPD. For instance, stakeholders must be clearly involved at every step. It must be stated how this was done. Mentor guidance can be provided to promote synthesisability. Next, practitioner-researchers should be encouraged to reflect on mismatches between theoretical approach and implementation. Mentors could suggest refinements to help resolve tensions. One way could be providing relevant summaries from OASIS. Practitioner-researchers could reflect on these, consult with mentors about their merits, and start planning action phases. By doing so, the mentor would provide knowledgeable guidance while still respecting teacher agency. The synthesist could investigate teachers’ decision-making. In the interests of synthesisability, these mentoring processes could be stated in the output documents- in the booklet foreword, or in detail in the QRS.

Conclusion

EAR for CPD has the potential to develop teachers’ interest in students’ experiences, environment and, in some cases, in academic literature by helping them notice gaps in their own practice. The benefits for professional learning and promoting habitual reflection for understanding ones’ social context and are clear. Furthermore, a critique of a QRS of TMTBLT has highlighted novel conceptual and methodological potential for addressing issues highlighted with EAR for CPD. Refocussing on a new research orientation- synthesisability, is recommended. Specific suggestions have been given for operationalising synthesisable EAR for CPD through QRS. By following these steps, a teacher could engage in exploratory professional learning and broaden the impact of their work. As has been done successfully in healthcare and TMTBLT, now too in EAR emancipatory reflection, technical and practical teaching improvements can continue while also generating synthesisable research findings. Both the teacher and the research team could leverage these into future classroom and career developments.

Acknowledgments

I would like to thank Mark Carver for his ever helpful advice and support.

Disclosure statement

No potential conflict of interest was reported by the author(s).