Learning outcomes of teacher professional development activities: a meta-study

Abstract

Former literature reviews suggested that if (student) teachers learn together in their professional development activities, professional development is enhanced. In the present literature review, we explored a variety of peer teacher professional development activities, conceptually divided into coaching, collaborating, and assessing activities. Specifically, we examined which learning outcomes could be achieved through participating in such activities. By means of a meta-study approach, we additionally reviewed methods, data collection, and theories used in the included studies. Findings showed outcomes were achieved in terms of teacher knowledge, teacher skills, and student learning. At the same time, many studies had methodological weaknesses: many self-report and self-constructed instruments hardly based on theory were applied. Based on the findings, we urge for more rigorous studies that can move the field forward.

Keywords:

• Teacher professional development
• meta-study
• learning outcomes

Teacher professional development (TPD) is considered to be an important approach for the improvement of quality of education (Borko and Putnam 1995; Coe et al. 2014; Desimone 2009; Hattie 2009). Many studies have explored what characteristics make TPD effective and concluded that TPD should focus on developing subject matter knowledge as well as pedagogical content knowledge, hold active and inquiry-based activities, span over a larger time period, and that teachers should work together with their colleagues (Desimone 2009; Putnam and Borko 2000; Van Veen, Zwart, and Meirink 2012). Examples of TPD wherein teachers learn with their colleagues are peer coaching (Ackland 1991) and communities of practice (Lave and Wenger 1991; Wenger 1998). In this article, we call such activities peer teacher professional development (PTPD) activities, and we will report on a review study that focussed on the learning outcomes of (student) teachers as a result of participating in such activities.

In this review study, a variety of PTPD activities are combined into a comprehensive overview. On a conceptual level and based on different functions of the activities (e.g. Little 1990) and group size, we distinguish three different types of PTPD activities: (a) coaching, wherein individuals support another individual, reciprocal or non-reciprocal, in achieving individual goals (e.g. peer coaching, peer feedback); (b) collaborating, wherein larger groups (than in coaching) collaborate on a shared goal and/or product (e.g. Community of Practice, teams, Lesson study); (c) assessment, wherein individuals assess another individual’s performance in either formative or summative ways (e.g. peer assessment, peer review). To obtain a comprehensive overview of outcomes of PTPD activities, this review study investigates research that has examined such activities with both pre-service and in-service teachers. As pre-service and in-service teachers arguably have different developmental needs and find themselves in different phases of the teaching career (Huberman 1989), results for both groups will be presented separately.

It is widely suggested that PTPD activities promote (student) teacher learning (Borko and Putnam 1995; Coe et al. 2014; Desimone 2009; Hattie 2009) and several reviews have suggested which characteristics of PTPD activities make them effective for teacher learning. However, they have explored the actual learning outcomes only to a limited degree. Moreover, former literature reviews focussed on one specific type of PTPD activities and on in-service or student teachers (e.g. Ackland 1991; Desimone 2009; Lu 2010; Van Veen, Zwart, and Meirink 2012).

Therefore, in the present literature review, we explore the outcomes of a variety of PTPD activities in a systematic way. More specifically we conducted a meta-study (Paterson et al. 2001). We choose this approach to synthesizing literature, because it includes a reflection on how methods, methodology, and theory-based decisions influenced studies’ findings (Paterson et al. 2001). In contrast to many other approaches for synthesizing literature a meta-study not only analyses and synthesizes findings, but it also analyses and synthesizes the research field itself with the intention to further develop theory, methodology, and methods. Therefore, our second aim is to examine theories, methodologies, and methods that were used throughout the studies with the intention to provide directions for future research on PTPD activities (cf. Paterson et al. 2001).

The research questions were: (a) What outcomes can be achieved through PTPD activities? (b) With which kinds of methodologies and methods were these findings revealed? (c) On which theories did the studies rely and what did they add to these theories? and (d) Taking the findings of research questions (a) through (c) together, what implications can be formulated for future studies?

Method

Meta-study is an approach for systematically synthesizing literature that aims to analyse and synthesize theory, methods, and results and to generate new knowledge about the phenomenon that was studied (Paterson et al. 2001). There are four steps that make a meta-study. First, a meta-data-analysis is conducted that analyses the findings of selected studies. Second, a meta-method is performed that explores the soundness of the selected studies as well as how the applied methods might have influenced the revealed findings. Third, a meta-theory is conducted that critically examines applied theoretical frameworks and emerging theory. Fourth, a meta-synthesis is performed that brings the three former steps together. In this step, the researchers “dig below the surface of what is currently understood, to draw on the most thorough analysis possible to deconstruct the validity of the ideas that are currently in favour, and to emerge with the kernel of new truth, a better kind of understanding, or a more socially responsible form of theorizing something” (Paterson et al. 2001, 111). In this section, we will further elaborate on how we conducted our meta-study.

Searching and selecting process

The search covered the period from January 1991 up to and including April 2013 and was limited to peer reviewed publications. To locate relevant studies, three search strategies were used. The first strategy was to explore two search engines namely EBSCO host and Science Direct. Within EBSCO host, we searched Academic search elite, Business source premier, Ejournals, PsychInfo, ERIC, and PsychArticles. A variety of search terms was used, such as peer coaching, collaborative learning, peer assessment, Community of Practice, and peer feedback. As alternatives for peer, we also used colleague, equal, and collegial. Each search term was additionally combined with teacher, student teacher, teacher education, and professional development separately.

Second, we manually searched eleven journals that specifically focus on (student) teachers’ learning and development and were therefore expected to contain relevant studies. These journals were, in alphabetical order, Action in Teacher Education, Asia-Pacific Journal of Teacher Education, European Journal of Teacher Education, Journal of Education for Teaching, Journal of Teacher Education, Teacher Development, Teacher Education Quarterly, Teachers and Teaching, Teachers College Record, Teaching and Teacher Education, and Professional Development in Education. Third, doctoral dissertations were searched. Additional criteria for inclusion of doctoral theses were that these should have been assessed by an assessment committee and should be available online.

The criteria for inclusion were as follows: (a) the article reported on an empirical study, (b) the participants were primary or secondary school teachers, special education teachers, or student teachers in primary, secondary or special education, (c) learning was explored in terms of process and activities in a formal setting, (d) participants were peers, and (e) the study examined outcomes of PTPD activities. Concerning criterion d, peers were defined as being of the same or similar status (cf. Ackland 1991). More specifically, this meant that student teachers should have the same amount of field experience and in-service teachers should have the same position in the collaboration process. We exemplify this delineation with one of the included studies. Vincent and Jones (2008) examined a coaching PTPD activity at a school that had provided each classroom with an interactive whiteboard. Two teachers were relieved of their teaching tasks in order to coach other teachers in using the interactive whiteboard. These two teachers did, like their colleagues, have no experience whatsoever in using interactive whiteboards. Therefore, they matched our definition of peers. If the two had had different experience in teaching with interactive whiteboards, they would not have been considered peers, and the study would have been excluded. If it was unclear whether the (student) teachers in a study met our definition of peers, the study would have been excluded.

Excluded also were studies that focussed on (a) dyads or triads of student teachers with cooperating teachers or teacher educators, (b) groups combining both student teachers and in-service teachers, and (c) groups wherein in-service teachers were more proficient on the contents of collaboration than other in-service teachers in the same group. We did include articles in which a group of (student) teachers (i.e., peers) was moderated by a teacher educator or researcher; however, we excluded articles where such a teacher educator or researcher actively participated in the learning process or activities.

While searching, the title and abstract of each article was judged against these five criteria. Articles were selected for further reading if they met at least four out of the five criteria. As a result, 902 articles were included for further reading, however, we could not retrieve the full texts of 45 out of these 902. Subsequently, the remaining 857 publications were judged against the five criteria and in order to be included in the meta-study, studies had to meet each single criterion. In case information lacked or was unclear, the study was excluded. Approximately two thirds were excluded because the (student) teachers did not match our definition of peers. In many of these studies detailed information was missing, such that we could not determine whether the (student) teachers were peers or not. In other cases, (student) teachers differed in field experience, veteran teachers coached novice or student teachers, or facilitators were too much involved. Approximately 200 other studies were not included because they did not meet one or more of the other criteria. Finally, 51 publications met each criterion and were included.

During initial reading, we discovered that some publications written by the same authors dealt with, probably or for certain, the same participants. This was the case, for instance, when a large amount of data had been collected and single publications focussed on different instruments. Paterson et al. (2001) suggested merging studies conducted on the same sample of participants. Therefore, we were able to analyse 44 unique datasets that were published in 51 articles.

Step 1: meta-data analysis

In the meta-data analysis, we focussed on outcomes that were achieved through PTPD. Data extraction sheets (Paterson et al. 2001) were created that provided an overview of the outcomes revealed in the included studies. In these data extraction sheets (i.e., Excel), the findings of studies were listed and organized.

To analyse the outcomes of PTPD found in the reviewed studies, we used Clarke and Hollingsworth’s (2002) interconnected model of teacher professional growth. This model describes four domains of change: the personal domain (i.e., changes at the personal level, such as knowledge, beliefs, and attitudes), the domain of practice (“place for experimentation”, which in all included studies was considered as changes in the teachers’ classroom behaviour), the domain of consequences (i.e., changes that go beyond teachers’ actions such as student learning), and the external domain (i.e., changes in resources and the school environment). According to this model, changes can occur within a single domain, in other words, a direct learning outcome occurs (Clarke and Hollingsworth 2002). Changes can occur through reflection (i.e., thinking) and through enactment (i.e., doing), resulting in domains affecting each other, in other words, a sequential learning outcome occurs. For example, the personal domain can influence the domain of practice because teachers try out (i.e. enactment) a didactical format. We have adapted the model slightly by dividing the external domain into a general external domain (e.g., changes in the whole school organization and leadership) and an intervention related external domain, in other words changes related to the PTPD activity and its characteristics, such as the role of facilitators and technology. As such we could examine whether teachers dealt differently (or not) with two different types of external sources (cf. Zwart 2007).

The outcomes were interpreted in two ways. First, direct learning outcomes regarding each single domain were revealed, for instance if teachers reported to have gained knowledge (personal domain). Second, patterns of changes (i.e., sequential learning outcomes) were revealed, wherein one domain was changed through either reflection or enactment via another domain. For instance, working in a wiki environment (intervention related external domain) enabled teachers to collaboratively synthesize (enactment) subject knowledge (personal domain; Biasutti 2011).

In a few studies, outcomes were revealed at the group level, for example a group of student teachers who developed materials for teacher-parent conversations (Huang, Lubin, and Ge 2011). The collective outcomes that were revealed in three studies, were listed in a separate data extraction sheet. Even though the model of Clarke and Hollingsworth (2002) does not focus on collective but rather on individual learning processes and outcomes, the collective outcomes were interpreted in line with the model.

The researchers had several meetings to discuss the interpretation and analysis of outcomes. In these discussions, the original studies and their descriptions of each outcome and its subcategories were compared to decide to which types of outcomes they belonged. Through these discussions, consensus was reached.

Step 2: meta-method

The meta-method phase explores how studies were conducted. Paterson et al. (2001) provided an overview of what can be analysed in this phase and how (such as the research questions, how researchers and participants are related, and data collection techniques), but also argued that each meta-method is different. We have used their suggestions to develop our own meta-method.

First, we mapped aims and research questions of the included studies and characterized the research questions in terms of the following categories: describing, explaining, comparing, evaluating, outcomes, designing, defining, and testing. As such, we were able to explore if aims and research questions were coherent. Second, we explored the design: was the study qualitative, quantitative, or mixed methods, was it conducted in a field or a laboratory setting, was it cross-sectional or longitudinal in nature, and was there a control group or not. We also examined researchers’ roles, for instance by examining whether they were involved in the intervention. Third, we explored the data collection methods used. For each variable, we mapped the method used (survey, interview, observation etc.), the source of this method, its reliability, and analysis method. Fourth, we examined which sampling procedures were applied. We also recorded the sample size. Finally, we mapped the setting of the study by summarizing the PTPD activity from each single study. For example, we determined whether the PTPD activity was collaborative, coaching, or assessment in nature and made an overview of the goals, duration and location of the intervention undertaken.

Each of these five aspects was recorded into data-extraction sheets (i.e., Excel and SPSS) for each single study. Subsequently, the data-extraction sheets were analysed and interpreted to perform the meta-method. Most information was categorized into either nominal or ordinal scales and frequencies were calculated.

Step 3: meta-theory

Meta-theory explores major paradigms and theories used (Paterson et al. 2001). Here, we created data-extraction sheets that mapped theories used and emerging theory (i.e., what does the study contribute to existing theory) for each study. Additionally, we mapped concepts used and their operationalization across studies. Finally, we evaluated these concepts using guidelines of Lenski (1988): (a) were concepts defined in such a way they could be measured, (b) were relations between concepts described, (c) how were these relationships described, and (d) was it, therefore, possible to falsify theory.

The data-extraction sheets were used to identify major paradigms and theories and shifts of paradigms or theories, and to explore the current status of the paradigm or theory.

Step 4: meta-synthesis

The final step of a meta-study combines the findings of the meta-data analysis, meta-method, and meta-theory. First, we determined with what kind of instruments the findings were originally revealed in each single study. Specifically, we established the nature of the instrument (quantitative or qualitative), the extent to which the instrument was a self-report measure or not, the degree of clarity with which the instrument was described, the source of the instrument and its reliability, and finally the evaluation of the theory by applying Lenski’s guidelines (1988). In other words, the data-extraction sheets of each former step were combined into one large data-extraction sheet. Second, we explored whether there were trends in the methods used, for example, to what extent the methods varied in terms of the nature of the instrument, the source of this method, and the evaluation of theory used.

Results

Meta-data analysis

In this section, we report findings that emerged from the 51 articles concerning outcomes of PTPD activities. Findings indicated that teacher peer learning contributed to teacher growth and that coaching and collaborating PTPD activities yielded outcomes on each of the four domains of the interconnected model of teacher growth (Clarke and Hollingsworth 2002). Only a few studies focussed on assessment activities and found outcomes for the personal and external domains. Table 1 shows how many outcomes for each of the four domains (i.e. personal domain, domain of practice, domain of consequence, and external domain) in the interconnected model of professional growth (Clarke and Hollingsworth 2002) were found and in how many studies they were revealed. About half of the studies demonstrated 113 different direct outcomes; the other half revealed patterns of changes. Of these 113 outcomes about 50% concerned the personal domain. For example, student teachers reported to have gained self-esteem for teaching (Shin, Wilkins, and Ainsworth 2007) and in-service teachers reported to have gained new ideas for teaching (Zwart et al. 2008). Examples of changes in the domain of practice were changes in using interactive whiteboards (Vincent and Jones 2008), the application of new teaching strategies (Rock and Wilson 2005) and adaptations of lessons to meet the needs of students (Estebaranz, Mingorance, and Marcelo 2010). Four studies revealed outcomes for the domain of consequence. For example, student teachers in Koc’s study (2011) reported to be more sensitive and emphatic to individual differences between their pupils. In their study on coaching in-service teachers in dealing with problematic behaviour of pupils, Stichter et al. (2006) demonstrated that pupils’ problematic behaviours significantly decreased after the coaching intervention. Additionally, three studies revealed outcomes with respect to the general external domain. For example, the schools in which teachers collaborated in teams made changes to school plans (Estebaranz, Mingorance, and Marcelo 2010) and student teachers were less inclined to skip lessons (Koc 2011).

Table 1. The number of outcomes found and the number of articles they were found in.

	Personal domain	Domain of practice	Domain of consequence	External domain general	External domain intervention related	Total
Collaborating peer teacher professional development (PTPD)
Number of outcomes	31	12	5	6	—	54
Number of articles	11	6	1	2	—	11
Coaching PTPD
Number of outcomes	20	24	12	—	—	56
Number of articles	7	9	3	—	—	13
Assessment PTPD
Number of outcomes	1	—	—	2	—	3
Number of articles	1	—	—	1	—	1
All PTPD
Number of outcomes	52	36	17	8	—	113
Number of articles	19	15	4	3	—	28

Note: The number of studies does not add up, because in several studies more than one outcome was present.

Many sequential outcomes were found, showing both the complexity and richness of participating in PTPD activities (Table 2). Three out of four domains (personal domain, domain of practice, and external domain) were influenced through reflection, but if teachers aimed to influence students (domain of consequence) enactment appeared a necessity (see Table 2). One example of this sequential outcome was found in Stichter et al.’s (2006) study. Of those teachers that participated in peer coaching to improve pupils’ disruptive behaviours, 75% reported that they felt that due to the peer coaching sessions their pupils’ behaviours indeed improved. Moreover, this was also confirmed by data on these pupils’ school records. Another example was revealed by Manouchehri (2001) who intensively followed two dyads of teachers while they were collaborating and coaching each other. In one of these dyads, the teachers developed a protocol for peer observation and feedback after one of their coaching sessions. This protocol led one of the two teachers to plan his instruction more extensively. In turn, the adapted instruction reinforced student learning in his class. No clear differences for direct or sequential learning outcomes were found between the types of PTPD.

Table 2. The number of sequential outcomes from external domain intervention-related and the number of articles they were found in.

	Collaboration PTPD		Coaching PTPD		Assessment PTPD		All PTPD
From external domain intervention via …	Number of outcomes	Number of articles	Number of outcomes	Number of articles	Number of outcomes	Number of articles	Number of outcomes	Number of articles
Reflection to …
… personal domain	30	13	20	9	8	1	58	24
… domain of practice	4	4	6	5	1	1	11	10
… domain of consequence	—	—	—	—	—	—	—	—
… external domain general	2	1	—	—	—	—	2	1
Total Reflection	36	14	26	9	9	1	71	27
Enactment to …
… personal domain	27	13	7	4	6	2	40	18
… domain of practice	8	4	14	6	—	—	22	10
… domain of consequences	2	2	4	2	—	—	6	4
… external domain general	3	2	—	—	2	1	5	3
… external domain intervention related	—	—	2	2	—	—	2	2
Total Enactment	40	16	27	10	8	2	75	28
Total sequential outcomes from external, intervention related domain	76	22	53	18	17	2	146	42

Note:

The number of studies does not add up, because in several studies more than one outcome was present.

Most learning patterns began in the intervention related external domain (Table 2). In 82% of all studies, 146 sequential outcomes were found that began in the external, intervention related domain (Table 2); the remaining 18% of studies had revealed direct learning outcomes only. For example, teachers involved in peer coaching said these discussions were beneficial as they helped them to gain insights into their colleagues’ views of teaching (Thijs and Van den Berg 2002). In such instances, the personal domain was influenced through reflection, starting in the intervention-related external domain. Another example stems from Manouchehri’s study (2001) which showed two teachers who collaborated and coached each other developed stronger beliefs about their own teaching through observations and their discussions. Here, the intervention-related external domain also influenced teachers’ personal domain, but through enactment. Similarly, many sequential outcomes were found that began in the external, intervention related domain and through either reflection or enactment influenced the domain of practice.

In a very small number of studies only a few sequential outcomes were found from the personal domain, the domain of practice, the domain of consequences, and the general external domain moving via reflection and enactment towards one of the other domains (Tables 3, 4, 5, and 6). One example was shown by Meirink (2007), who studied the learning processes of teams of in-service teachers while they were adapting their teaching practices towards students’ active and self-regulated learning. Teachers, whose beliefs (personal domain) were in alignment with this innovation, adapted (enactment) their teaching practices (domain of practice) in such a way that their students were given more responsibility for their own learning.

Table 3. The number of sequential outcomes from personal domain and the number of articles they were found in.

	Collaboration PTPD		Coaching PTPD		Assessment PTPD		All PTPD
From personal domain via …	Number of outcomes	Number of articles	Number of outcomes	Number of articles	Number of outcomes	Number of articles	Number of outcomes	Number of articles
Reflection to …
… domain of practice	1	1	3	1	—	—	4	2
… domain of consequences	—	—	2	1	—	—	2	1
… external domain general	—	—	—	—	—	—	—	—
… external domain intervention related	—	—	—	—	—	—	—	—
Total Reflection	1	1	5	1	—	—	6	2
Enactment to …
… domain of practice	1	1	2	1	—	—	3	2
… domain of consequences	—	—	—	—	—	—	—	—
… external domain general	—	—	—	—	—	—	—	—
… external domain intervention related	2	2	2	1	—	—	4	3
Total Enactment	3	2	4	1	—	—	7	5
Total sequential outcomes from personal domain	4	3	9	1	—	—	13	5

Note: The number of studies does not add up, because in several studies more than one outcome was present.

Table 4. The number of sequential outcomes from domain of practice and the number of articles they were found in.

	Collaboration PTPD		Coaching PTPD		Assessment PTPD		All PTPD
From domain of practice via …	Number of outcomes	Number of articles	Number of outcomes	Number of articles	Number of outcomes	Number of articles	Number of outcomes	Number of articles
Reflection to …
… personal domain	—	—	1	1	—	—	1	1
… domain of consequences	—	—	2	2	—	—	2	2
… external domain general	—	—	—	—	—	—	—	—
… external domain intervention related	—	—	—	—	—	—	—	—
Total Reflection	—	—	3	2	—	—	3	2
Enactment to …
… personal domain	—	—	1	1	—	—	1	1
… domain of consequences	—	—	1	1	—	—	1	1
… external domain general	—	—	—	—	—	—	—	—
… external domain intervention related	—	—	—	—	—	—	—	—
Total Enactment	—	—	2	1	—	—	2	1
Total sequential outcomes from domain of practice	—	—	5	3	—	—	5	3

Note: The number of studies does not add up, because in several studies more than one outcome was present.

Table 5. The number of sequential outcomes from domain of consequences and the number of articles they were found in.

	Collaboration PTPD		Coaching PTPD		Assessment PTPD		All PTPD
From domain of consequences via …	Number of outcomes	Number of articles	Number of outcomes	Number of articles	Number of outcomes	Number of articles	Number of outcomes	Number of articles
Reflection to …
… personal domain	—	—	2	1	—	—	2	1
… domain of practice	—	—	—	—	—	—	—	—
… external domain general	—	—	—	—	—	—	—	—
… external domain intervention related	—	—	—	—	—	—	—	—
Total Reflection	—	—	2	1	—	—	2	1
Enactment to …
… personal domain	—	—	1	1	—	—	1	1
… domain of practice	—	—	—	—	—	—	—	—
… external domain general	—	—	—	—	—	—	—	—
… external domain intervention related	—	—	—	—	—	—	—	—
Total Enactment	—	—	1	1	—	—	1	1
Total sequential outcomes from domain of consequences	—	—	3	2	—	—	3	2

Note: The number of studies does not add up, because in several studies more than one outcome was present.

Table 6. The number of sequential outcomes from external domain and the number of articles they were found in.

	Collaboration PTPD		Coaching PTPD		Assessment PTPD		All PTPD
From external domain via …	Number of outcomes	Number of articles	Number of outcomes	Number of articles	Number of outcomes	Number of articles	Number of outcomes	Number of articles
Reflection to …
… personal domain	—	—	1	1	—	—	1	1
… domain of practice	—	—	—	—	—	—	—	—
… domain of consequences	—	—	—	—	—	—	—	—
… external domain intervention related	1	1	—	—	—	—	1	1
Total Reflection	1	1	1	1	—	—	2	2
Enactment to …
… personal domain	—	—	—	—	—	—	—	—
… domain of practice	—	—	1	1	—	—	1	1
… domain of consequences	—	—	—	—	—	—	—	—
… external domain intervention related	—	—	2	1	—	—	2	1
Total Enactment	—	—	3	1	—	—	3	1
Total sequential outcomes from external domain	1	1	4	2	—	—	5	3

Note: The number of studies does not add up, because in several studies more than one outcome was present.

There were three studies, all with the collaboration PTPD activities, revealing outcomes at the group level. In each of these studies, groups of (student) teachers worked together to develop a shared product. For example, the student teachers in Nicholas and Ng’s (2009) study made curriculum materials for science education that were of high quality. Huang, Lubin, and Ge (2011) compared two different groups (i.e. constructivist collaborating vs. traditional lectures) and concluded that the documents created by students in the constructivist setting were more specific, coherent and thoughtful than in the other settings. These outcomes show collective results for the domain of practice. Teachers in Shriki and Movshovitz-Hadar’s (2011) study developed lesson plans for mathematics in an online wiki environment. This led to a lively discussion about ownership and copyrights of the materials. This can be considered as a collective outcome for the personal domain, in which attitudes and opinions of the whole group were shared.

Meta-method

In this section, we report findings that emerged from the 44 different datasets, unless we specifically refer to one of the 51 included articles, for instance when two articles were published on the same data-set, but each focussed on different instruments. Four studies were published in the 1990s, 31 in the 2000s, and 16 in the 2010s. On average, articles had a length of 16.43 pages (standard deviation [SD] = 5.62, minimum = 6, maximum = 35). A large variety was found in terms of the settings of the studies.

The studies were conducted throughout the whole world. The United States was represented most: 18 studies were situated there. Europe was represented by 12 studies. More specifically, six studies originated from the Netherlands, two from the United Kingdom, two from Italy, one from Finland and one from Spain. Six studies were conducted in Asia. Two of these studies were conducted in Turkey, two in Hong Kong, and one in Egypt and one in Israel. Six studies were conducted in Australia and one in New Zealand. One study originated from Africa, more specifically from Botswana.

In 17 studies, (student) teachers participated in coaching activities, in 25 studies they participated in collaboration activities, and in two studies in assessment activities (see Table 7 for details). Twenty-nine studies were situated in face-to-face settings and 12 studies used online, mostly asynchronous, learning environments. Three other studies applied a blended approach.

Table 7. Overview of settings of interventions.

	Student teachers (n = 28)	In-service teachers (n = 15)	Student and in-service teachers in different groups (n = 1)	Total (n = 44)
Setting: location
Face-to-face	17	12	—	29
Online	8	3	1	12
Blended	3	—	—	3
PTPD activity
Coaching	11	6	—	17
Assessing	2	—	—	2
Collaborating	15	9	1	25
Cluster of facilitation
1	3	3	—	6
1–2	14	5	—	19
2	1	1	—	2
2–3	1	—	—	1
3	7	4	—	11
Unknown	2	2	1	5

On average, 51 participants were included within one study (SD = 93.26, minimum = 3, maximum = 616). Sampling methods were unclear in two studies. In 31 studies, the sampling was convenient and in many of these studies the researchers were probably teacher educators and the participants most likely their student teachers. In six studies, an instrument, typically a survey, was distributed among members of PTPD groups (e.g. an online forum; Hanewald and Gesthuizen 2010; Schuck 2003) and those that replied formed the sample. In five studies, the sample was recruited through contacting districts or schools and those that volunteered, and in some cases met certain criteria, were included.

Table 7 focusses on the role of the facilitator. To facilitate the interpretation of findings, we grouped the facilitators’ tasks into three main clusters. In cluster 1, no facilitator was presented. In cluster 2, the facilitators’ task was to provide technical and/or organizational support. In cluster 3, the facilitators’ task was to support the participants in their processes, for instance by providing guidelines on collaboration. Additionally, combinations of clusters were also found. Cluster 1–2 refers to no facilitator being present during the participants’ learning processes, however, there was technical and/or organizational support available. In cluster 2–3 both technical and/or organizational support and support on the learning processes was available, sometimes with more than one facilitator. Table 7 shows that in the majority of studies, the facilitators were not involved in the process, but did provide technical and/or organizational support. Cluster 3 was present in about one in five studies. A major difference was found between studies with student teachers, where mostly no facilitator was involved, and studies with in-service teachers, where in about half of the interventions no facilitator was involved.

The duration of the interventions varied from two weeks to two years. Some interventions took a whole school year (five interventions of 40 weeks). Additionally, many studies spanned a course (ranging from seven weeks to 20 weeks, in 23 interventions) and in seven studies it was unclear what the duration was.

The group size wherein the PTPD activities were conducted varied throughout the studies, however, in the majority of interventions the group size ranged from two to four participations (i.e. in 26 studies). In eight studies, the group size varied between four and 10 participants and in six studies a whole class of student teachers or a whole team of in-service teachers participated. In two studies, group size was 150 and 616. In these studies, a whole online discussion forum (Schuck 2003) and a mailing list (Hanewald and Gesthuizen 2010) were the context of the study.

In 16 studies, no research questions were reported and in one of these studies (Vincent and Jones 2008) clear goals were also lacking. Of the other studies, the link between goals and research questions was explored and we found that in 13 studies the link was partially visible, in seven studies it was clearly visible, and in eight studies the link was crystal clear. On average, studies formulated two research questions (SD = 1.69, minimum = 0, maximum = 7). Different types of research questions asked were: (a) descriptive: 13 questions in 13 studies, e.g. How do student teachers construct meaning …, (b) explaining: six questions in six studies, e.g. Which factors influence …, (c) comparing: two questions in two studies, e.g. To what extent does x differ between group A and group B, (d) evaluative or valuing: 14 questions in 14 studies, e.g. What are teachers’ opinions about this intervention, (e) outcomes: 15 questions in 15 studies, e.g. What changes have been taking place…, (f) designing: three questions in three studies, e.g. What can be improved regarding this intervention, (g) defining: one question in one study, e.g. What are the categories of responses …, and (h) testing: questions in two studies, e.g. Will X be improved in setting A, compared to setting B. Evaluative or valuing research questions were often combined with outcome questions. Two studies that posed comparing questions applied control groups; other studies that used control groups did not ask such comparing questions.

The overall design of the studies varied throughout. About two-thirds (i.e. 28 studies) of the included studies applied a qualitative approach. A quarter of them (i.e. 13 studies) used a mixed-method design and 6% (i.e. three studies) was of quantitative nature. All studies were conducted in a field setting. Two studies (El-Deghaidy and Nouby 2008; Grion and Varisco 2007) collected pre- and post-intervention data. Eighteen studies (41%) collected post-intervention data only; six studies (14%) collected data during interventions; 10 studies (23%) collected data during and after interventions; and eight studies (18%) collected data before, during, and after interventions.

The majority of studies (87%) did not apply a control group. In four studies that used a control group (Burron, James, and Ambrosio 1993; El-Deghaidy and Nouby 2008; Huang, Lubin, and Ge 2011; Stichter et al. 2006), the control group followed a more traditional learning trajectory, while the experimental group was involved in more innovative interventions that were based on social constructivism. In two other studies (Olson 2007; Sluijsmans, Brand-Gruwel, and van Merriënboer 2002; Sluijsmans et al. 2004), the control group was used to explore outcomes compared to the experimental group of for instance training (Sluijsmans, Brand-Gruwel, and van Merriënboer 2002; Sluijsmans et al. 2004).

The researchers were engaged in the interventions in several cases. In 20 studies, the researchers acted as facilitators and in four studies they were present but their role during interventions was less active. In 18 studies, the researchers were hardly or not involved in the interventions. Moreover, from the articles we could deduce that in nine studies the facilitators were also the researchers and authors. In nine studies this was not the case and from 26 articles it was unclear whether the researchers had also been the facilitators. Such involvement might lead to bias. In hardly any article, the authors acknowledged this possible bias. In a few studies (Hamilton 2013; Koc 2011) researchers asked an independent colleague for reliability checks.

Additionally, a large variety of data collection methods was used, yet the majority of these were self-report measures and many of the instruments were developed by the researchers themselves. In many cases, information about the development of instruments, the analysis, and reliability was lacking. Together, the included studies applied 108 instruments to collect data. On average, studies used 2.38 different instruments, with a minimum of one and a maximum of six instruments. More specifically, instruments used were (the amount is shown between brackets): single surveys (24), observations (20), interviews (14), products or tests made by the participants (13), reflection reports (10), student information (6), researchers’ field notes (4), pre- and post-surveys (4), focus groups (4), participants’ logbooks (4), fidelity checks (i.e. an instrument that explored if the intervention was implemented as intended; 2), peer coaching or review forms (2), and rating forms (1).

Similar to exploring the connection between goals and research questions, we examined whether the instruments used in a study matched the goals and/or research questions posed. In three studies the connection between instruments and goals and/or research questions was completely unclear; these studies had also been unclear about the link between goals and research questions. In four studies the link between instruments and goals and/or research questions was partially clear. In 19 studies this link was clear for most instruments and the instruments would support addressing goals and answering research questions for the most part. In 18 studies the link was absolutely clear and the instruments would support addressing all goals and answering all research questions. We found a significant relationship between the clearness of the link between goals and research questions and the clearness of the link between instruments and goals and research questions (r = 0.52, p = 0.00). This means that if one of the links was clear, the other was also clear. We also found a significant relationship between the length of the article and the clearness of the link between instruments, and goals and research questions (r = 0.47, p = 0.00). This indicates that longer articles were clearer about the link between their goals and research questions and the instruments they used than shorter articles.

Furthermore, we examined three characteristics of the instruments used. The first was how clear and specific the authors had described the instruments. Did they provide a reasoning for using the specific instrument and did they include items or questions from the instrument? The second was the source of the instrument: was it self-developed or did it originate from another or former study? Third, we investigated reliability measures of the instruments. Were reliability measures included, which ones, and was that sufficient?

To explore how specific the information about the instruments was we used a six-point scale. Findings show that for 16% of the instruments such specific information was lacking; authors mentioned that they used a specific instrument without any further explanation. Next, for 43% of the instruments a description of and/or reasoning for the specific instruments was included, and the information of 67% of the instruments was in between these extremes. Concerning 9% of the instruments the description was such that it was possible to deduce items or questions and for 10% of all instruments examples of such items or questions were included in the article. Finally, 13% of the instruments was completely included in the article (in tables, figures, or appendices).

The majority of instruments (80%) was constructed by the researchers themselves. 18% of the instruments was based on former literature or validated instruments were used. Two instruments were partly self-constructed and partly based on former literature. One instrument (a test; El-Deghaidy and Nouby 2008) was based on existing items from a course test-bank.

Next, we examined reliability measures of the used instruments and found that in 57% of the instruments no such information was provided. Concerning 12 instruments (11%) several researchers were involved and through discussions they reached agreement. This method was mostly applied in coding observations and interviews. In 29 cases, reliability measures were given that met standards for reliability measures. For one instrument an actual reliability measure was given, but this did not meet standards for reliability measures. Finally, in three cases reliability measures were also provided but some did and some did not meet reliability criteria. This was often the case for surveys that applied more than one scale and reliability measures differed between the scales.

Meta-theory

The majority of studies was based on either social cultural theory (citing for instance Vygotsky 1978), Communities of Practice (citing for instance Wenger and Snyder 2000), or social constructivism (citing for instance Jonassen 1999). Some of the studies implied or explicitly stated that they were based on two or even three of these theories. Other studies were based on Computer-supported Collaborative Learning (CSCL; Strijbos, Kirschner, and Martens 2004) or Computer-mediated Communication (CMC; Walther and Tidwell 1995). Still other studies were based on other forms of collective learning, such as peer coaching. This is not so much of a theory itself, but it resonates with principles of social cultural theory, Communities of Practice, and social constructivism. The final group of studies was based on a variety of theories and concepts, such as reflection (e.g. Amobi 2005) or approaches to learning (e.g. Lee and Baek 2012). One study (Estebaranz, Mingorance, and Marcelo 2010) was completely unclear on what theory it was based, and in several studies the theory was not explicitly stated.

Based on the clearness of theories that the studies were based on, emergent theory (i.e. what a study contributes to existing theory), and the guidelines of Lenski (1988), we distinguished three groups of studies in terms of the quality of each study’s theoretical underpinnings. The first group of 16 studies (36%) could be considered as poor with respect to its theoretical underpinnings: the used theory could not be falsified. This meant that these studies were often based on just one concept or in a few cases such a concept even seemed lacking. Additionally, half of these studies repeated their findings in the discussion sections and gave practical implications but did not explain what the contribution of their study was to former literature or theory. The second group held 21 studies (48%) and the quality of these studies’ theoretical underpinnings was intermediate: the used theory could be falsified to a certain degree. Generally, these studies contained more than one concept and their discussion sections went further than just repeating findings. The concepts researched were described more clearly than in the first group. Additionally, these studies described the relationship between concepts in a fairly explicit way. The third group of seven studies (16%) were of good theoretical quality: the theory used could be falsified. These studies also explored more than one concept and the relation between concepts was overall explicitly described, either in text or in figures (e.g. a conceptual model or structural equation model). Additionally, these studies were building or expanding theory in their discussion sections.

Theoretical underpinnings used to study PTPD seemed fairly stable over time: there were no indications that the theory a study was based on related to year of publication. Some studies were situated in reforms or innovation processes. For instance Meirink (2007) and Zwart et al. (2008, 2009) both studied teacher learning in a reform in the Netherlands about implementing active, self-regulated student learning in secondary education and Vincent and Jones (2008) who studied the implementation of interactive white boards. However, despite such a different context compared to other studies, they all used about the same theories.

Meta-synthesis

In this section, we elaborate on the final step of the meta-study: the meta-synthesis. In this final step, the findings of the meta-data analysis, meta-method, and meta-theory were brought together. As such, we were able to explore which outcomes were revealed in the original studies with what kinds of methods, their quality, and theoretical underpinnings.

The studies showed PTPD activities were effective for (student) teachers’ learning. They gained knowledge and insights, they developed attitudes, and came to realize their strengths and weaknesses as a teacher. They were able to change teaching practice. To a lesser extent studies also showed student learning and behaviour as well as the school environment could be improved. However, findings were mostly revealed via self-report measures and instruments were mostly self-constructed. In other words, from a methodological point of view the evidence for learning outcomes of PTPD has a rather weak basis.

Studies hardly focussed on collective learning outcomes. Even though the theoretical underpinnings of the majority of studies were similar – based on social learning – the data collection measures hardly explored collective outcomes. Studies on PTPD were based on similar kinds of theory, which seemed to be limited in the number of theories as well as in the depth of using those theories. Moreover, because of this limited use of theory, the development of new theory is lacking. In other words, research on PTPD activities is stagnant and of questionable strength.

Discussion

This literature review was approached with a meta-study (Paterson et al. 2001), in which we explored which outcomes could be achieved through participating in PTPD activities. Findings firstly showed PTPD activities enhance outcomes at the personal level, the classroom level, at the student level, and to a lesser extent at the school level. Second, characteristics of the interventions (external, intervention-related domain) influenced the outcomes. Finally, only if teachers enacted rather than reflected (Clarke and Hollingsworth 2002), the student level could be influenced. By means of the meta-study approach, we also examined the methods, data collection, instruments, their origins and reliability, theories used and emerging theories from the included studies.

The interconnected model of teacher professional growth (Clarke and Hollingsworth 2002) was used as a framework to analyse the outcomes of PTPD activities. Our findings could be well-framed within this model and showed that teachers can enter learning at any domain (Zwart et al. 2007). The interconnected model of teacher growth was found to be a useful model for interpreting the individual learning outcomes, especially for the direct outcomes. To interpret the sequential learning outcomes was more difficult because the original, raw data from participants was not available. At the same time, using Clarke and Hollingsworth’s (2002) model made it possible to reveal such sequential learning outcomes via reflection and enactment, which would not have been found if we had used for example Kirkpatricks’ (1994) four levels (reaction, learning, behaviour and results) for evaluating training programmes. Unfortunately, many studies examined outcomes of PTPD activities and did not explicitly connect these to the characteristics of those activities. Therefore, a challenge for future research is to explore how certain characteristics of PTPD activities elicit certain outcomes. For example, which type of PTPD activity is most suited for outcomes at the personal level or at the student level?

The interconnected model of teacher professional growth (Clarke and Hollingsworth 2002) was also applied to analyse the few collective outcomes of PTPD activities. Future research can explore such outcomes at a group level as only three out of 44 unique studies did so. An example of such a study is Doppenberg, Bakx, and den Brok (2012) who interviewed seven teams of two primary school teachers and their school leader. They revealed that teachers individually and collectively learned from each other in a variety of settings, however, the intensity differed between settings. Another suggestion for future research is to develop a theoretical lens for interpreting collective outcomes. In the present review, collective outcomes were interpreted with Clarke and Hollingsworth’s (2002) model and the domains were considered from the group’s perspective. While this was possible and the researchers agreed on this interpretation, there might be other aspects to consider for such a theoretical lens.

The meta-synthesis step led to the conclusion that research on PTPD activities is not making much progress. What is needed for future research is the development of new or application of other theories to PTPD activities. Such theories will also bring other data collection strategies and methods. Such theories can be found for example in psychology or human resource development literature, such as research on professional expertise (Van der Heijden 2000). Professional expertise is a construct consisting of several dimensions, such as having knowledge needed for one’s field of expertise, having meta-cognitive skills (knowing what you do and do not know and how to develop), and being flexible. Another example of such a theory is Fishbein and Ajzen’s (2010) theory of planned behaviour that reasons which factors influence the intentions to behave. This theory distinguishes proximal and distal factors, respectively factors that are closely connected to the behaviour such as attitudes, and factors that act at a larger distance such as the organizational culture. Another suggestion for deepening research on PTPD would be to develop new methods and methodologies which support the development of theory. For example, Leeferink et al. (2015) used a narrative analysis to explore the learning processes of student teachers in their field experiences. Their analysis was an in-depth exploration of student teachers’ learning processes by which they revealed a variety of factors influencing this process. Additionally, using this methodology Leeferink et al. developed several chains of student teachers’ learning processes, by which they showed the complexity of these learning processes. Future research with strong methodological and theoretical lenses is needed to explore the complexity of (student) teacher learning processes, as well as research focussing on collective learning outcomes.

Seven of the reviewed studies were of high quality based on the meta-theory, four of which were connected to doctoral theses. These studies used instruments that were embedded into theory and/or they used data-analysis methods that had a theoretical lens. Additionally, they used triangulation in their methods and some of their instruments had been validated in former studies. Findings of these studies confirmed insights from the other included studies: PTPD activities promote changes in the personal domain, domain of practice, domain of consequences, the external domain, and – to a lesser extent – outcomes at the group level. Additionally they showed characteristics of interventions (external, intervention-related domain) can influence changes within the four domains, through enactment and reflection.

Therefore, based on these seven studies, for future research we recommend relying less on self-report and self-constructed instruments and using more than one instrument (triangulate). If researchers use self-constructed instruments, they can provide more information about its construction and theoretical basis. Additionally, we recommend paying more attention to reliability and validity issues, both in the process of actually doing a study and in the process of peer review used in many Journals, where editors and reviewers can address these issues, asking for such information. Moreover, Journal editors may wish to reconsider word limitations. We found that the longer an article was, the clearer information on data collection and analysis became, as well as their connection to goals and research questions.

This review showed by revealing sequential learning outcomes, that characteristics of PTPD activities guided learning. The question of which of these characteristics make PTPD activities effective in terms of that learning, cannot be answered with this review. Other literature reviews have focussed on this question (e.g. Coe et al. 2014; Desimone 2009; Timperley 2008; Van Veen, Zwart, and Meirink 2012), and showed TPD activities should span at least one year, focus on gaining (pedagogical) content knowledge and encompass inquiry activities wherein participants collectively and actively engage in linking practice and theory. Additionally, expertise should be brought in from outside of the school and the aim and content of the activities should cohere with school and/or national policy.

One practical implication is to develop PTPD activities in line with these characteristics – also as these characteristics were not present in all studies (e.g. the time span was shorter than recommended, especially for student teachers). Another practical implication is that (student) teachers should participate in PTPD activities as they were shown to be beneficial to (student) teachers, schools, students, and teacher education. As collaborative activities yielded relatively more outcomes for the personal domain and for the external domain (Table 1), it is suggested that teachers participate in such activities if knowledge gain, changing attitudes, or making plan for the whole school are the aim of learning. Since coaching activities yielded relatively more outcomes for the domain of practice and domain of consequences (Table 1), it is suggested that teachers participate in such activities if outcomes such as instruction skills or student learning are aimed for. Moreover, teachers should be given ample opportunities to reflect on and act on the contents of the activity as our review showed that these indirect processes are important and influence outcomes. If outcomes are aimed for on the student level, teachers need to act rather than reflect. Finally, even though PTPD can yield positive outcomes it is important to realize that the manner in which these interventions are designed and how teachers work together to a large degree affect the outcomes of TPD.

Disclosure statement

No potential conflict of interest was reported by the authors.

Funding

The work was supported by Nederlandse Organisatie voor Wetenschappelijk Onderzoek [grant number 411-12-202]