Enhancing teachers’ STEM understanding through observation, discussion and reflection

ABSTRACT

Effective teaching of STEM has become a significant concern for teachers’ professional development (PD). Many teachers are not familiar with STEM strategies and do not possess the disciplinary knowledge or pedagogical strategies demanded by STEM teaching. The purpose of the study reported in this paper was to explore the path and model of Chinese teachers’ professional development about STEM understanding based on their participation in the Zhejiang-Indiana STEM summer programme. This programme capitalises on strategies referred to as the observation-discussion-reflection (ODR) framework. The participants were 82 teachers who specialise in science, technology, or mathematics and who attended the training activities. There were two weeks of training interventions. Statistical analysis of two groups of teachers’ pre-test and post-test, as well as the comparative analysis of two groups, were conducted. The changes in teachers’ STEM discipline knowledge as well as their abilities and attitudes were investigated. Overall, the findings indicate variations in how teachers understand the different components of STEM knowledge with integrated knowledge requiring the most attention for further professional development.

KEYWORDS:

• STEM literacy
• stem education
• classroom observation
• discussion in group
• video reflection
• ODR

Introduction

Since the Science, Technology, Engineering, and Mathematics (STEM) Education Integration framework (originally referred to as SEM& T) was proposed by the National Science Board (NSB) in the USA in 1986, STEM education has become a widespread agenda globally. Policy makers and educational leaders have argued that the key to future prosperity is improving STEM teaching and learning opportunities (National Academy of Sciences, et al. 2007). In 1996, the National Science Foundation (NSF) reflected on STEM education in a ten-year review and summarised that focusing on ‘the needs of college students in various two-year and four-year colleges in the United States’ requires ‘cultivating K-12 education system SEM&T’s faculty issues’. The Innovative America: Developing Science, Technology, Engineering and Maths Agenda proposed by the US Governors Association in 2007 not only explains the background, current status, problems and strategies of STEM implementation but also points out the challenges teachers face in adopting pedagogical strategies that support effective learning of STEM. One concern is that teachers have limited background in STEM and disciplinary orientations demanded by STEM. Many educational systems around the world currently face similar challenges, and different educational systems advocate models of STEM teachers’ professional development (Hayden et al. 2011; Hsu and Yeh 2020; Taningco, Mathew, and Pachon 2008).

In this paper, recent research literature on STEM education is reviewed, particularly in relation to characterising STEM literacy and the role of teachers’ professional development in its achievement. The research literature illustrates the components of STEM teaching and learning, and the demands that STEM places on secondary school teachers. The discussion is situated in the context of China where a professional development project was carried out with the objective of improving teachers’ STEM knowledge and attitudes towards STEM. Findings from the empirical study is reported focusing on the impact of a professional development intervention.

Review of literature

One goal of STEM education is to cultivate STEM literacy for all learners (Bybee 2010, 2013). Different scholars have different views on the characteristics and definition of STEM literacy. Bybee (2010) argues that STEM literacy includes conceptual understanding, procedural skills and abilities for individuals to address STEM-related personal, social, and global issues. Some researchers elaborated on the definition of STEM literacy as pertaining to skills that are demanded in the 21st century which include problem-solving skills, social communication skills, technology and engineering skills, system skills, and time, resource, and knowledge management skills as the key competencies for STEM workers (Morrison 2006). They proposed that STEM literacy is a compound literacy, which is the integration and expansion of the four specialisations (i.e. scientific, technical, engineering and mathematical literacy). According to Zollman (2012), the ultimate goal of STEM education should change from learning STEM literacy to using STEM literacy for learning. The implications of research and policy literature is that STEM literacy can be divided into three dimensions: STEM knowledge, STEM ability, and STEM attitude.

STEM literacy requires students and future citizens to develop the competencies to apply basic content and practices in STEM disciplines within the situations they encounter. In order to meet the challenges in 21st century, different scholars have given different opinions on the abilities that students need to have. There are many similarities between STEM literacy and STEM competencies. However, STEM competencies focus more on the skills needed for future careers whereas STEM literacy is more comprehensive. The promotion of STEM literacy requires explicit education, which demands teachers’ STEM literacy. Consequently, teachers’ understanding of STEM affects students’ achievement (Yoon et al. 2007; Capraro 2013). In order to achieve effective STEM teaching, teachers must have content knowledge (i.e. four disciplines of STEM), pedagogical content knowledge (PCK) of STEM content and pedagogical strategies (e.g. problem-solving, scientific inquiry, engineering design). Furthermore, their attitudes are important in defining how they will approach STEM teaching (Eckman, Williams, and Silver-Thom 2016). Moreover, to develop students’ STEM literacy through teaching, teachers must be informed of STEM literacy and teaching ability, which requires high-quality professional development. However, many teachers are not qualified for STEM teaching and have limited understanding of STEM (Czerniak and Johnson 2014). In the case of science teachers, there is a finite awareness of technology and engineering (National Governors Association 2011; National Research Council 2011; Thibaut, et al. 2018).

Approaches to improvement of teachers’ understanding of STEM

Compared to the traditional transmission model of teaching, STEM education places higher demands on teachers and requires teachers to teach knowledge skills as well as skills such as critical thinking and problem-solving (Corlu, Capraro, and Capraro 2014). However, a significant proportion of training currently received by teachers internationally includes only one subject, such as science or mathematics. Hence, there is an urgency for effective STEM PD to help teachers develop STEM literacy and improve STEM understanding (Ye and Yang 2018). Teacher training programmes can include mentoring, workshops, coaching, support groups, and online training (Pfeffer and Sutton 2000). Most PD courses focus on pedagogical content knowledge (PCK) and the attitudes of teacher participants (National Research Council 2015). Effective PD requires training with concrete tasks of teaching, assessment, discussion, observation and reflection (Darling-Hammond and McLaughlin 2011). Postholm (2008) proposed that reflection is the key to teachers’ learning and development of teaching experience and showed that teachers can reflect before action, in action and on action, connecting theoretical concepts with their teaching practice (p.17).

Furthermore, Shernoff et al. (2017) argued that improving teachers’ STEM understanding should give in-service teachers opportunities to observe out-of-district teachers and have more time for teacher collaboration and see examples of effective lessons. Therefore, using observation as a form of professional development improves teaching practices and professional competence (Alshehri 2019). Moreover, teachers’ discussion plays an important role in teachers’ professional development training and provides opportunities for in-depth peer-to-peer interaction. The approaches to effective STEM teacher PD include continuous, participatory teacher training, theoretical study, field trips and listening courses, and STEM professional training requires teachers to go through scientific research and engineering design processes before teaching (Crippen, Biesinger, and Ebert 2010; Brown and Crippen 2018). The observation-discussion-reflection (ODR) approach, as an effective way to promote professional development in STEM education, can keep teachers active in personal learning and understanding STEM and provide a framework within which to share their own discussion, reflections and observations.

The purpose of the ODR framework is to extract the main approaches of PD training that affect teachers' STEM understanding. The most direct and basic approach to improve teaching practices is to allow teachers to observe high-quality courses (American Association for the Advancement of Science, 2012). Then, observers and participants discuss both students’ learning ways and teachers’ instructional strategies through the STEM observation forms, lesson design, lesson core content (Borko, et al. 2011). Reflection helps foster meaningful discussions and potentially improve teachers’ pedagogy. All groups have experienced the process of ODR, but the specific content and task requirements may be different depending on the cohort of teachers. ‘The Chinese Lesson Study or Parallel Classroom Study’ refers to a type of professional development that includes certain topics, lesson plans, classroom observation, collaborative discussion, post-lesson debriefing and reflection (Yang and Ricks 2012; Huang, et al. 2014).

STEM literacy and professional development of Chinese teachers

Despite a wealth of reports on STEM education and PD in STEM teaching in China, there is paucity of research on how Chinese teachers are being prepared for improving STEM teaching and few studies on teachers’ STEM literacy. One of the few empirical studies on the status of STEM education in Chinese primary and secondary schools, Huang, et al. (2020) reports that STEM teachers are mostly young and middle-aged and that their understanding of STEM needs to be deepened. For behavioural intention and behaviour attitude, the performance of science and engineering teachers is better than that of liberal arts teachers. In terms of self-efficacy and output quality, engineering teachers are significantly further along. Furthermore, focusing on the status of Zhejiang teachers, the same study showed that the understanding of the discipline concept (four disciplines of STEM) is not balanced and that teachers do not have a sufficient understanding of mathematics in STEM. Problem-solving ability, technical application and mathematical modelling were also limited. Moreover, the study of our team (Huang, et al. in press) showed that the overall situation of STEM literacy of teachers in Zhejiang Province urgently needs to be improved. Problems of preservice teachers’ STEM literacy, which includes knowledge understanding, problem solving ability and STEM attitudes related to different majors, grades and genders, are highlighted. Similar problems existed in the survey of in-service teachers; they performed well in the areas of disciplinary knowledge, but they lacked problem solving (PSA) and STEM career interest.

The professional development of STEM teachers in Zhejiang Province aims at preservice teachers and in-service teachers, including expert lectures, workshops, exchanges and exhibitions among teachers after practice and organises to visit STEM schools in the US. During 2017–2019, ZIPCP was aimed at Zhejiang primary and secondary school teachers for two weeks during summer vacation. In 2019, the STEM education training conference and Zhejiang STEAM Education Conference, China-US-Canada STEAM Advanced Training Conferences were held for improving teachers’ STEM theory and teaching practice, especially STEM literacy and PCK for STEM. The research and practice of STEM education in Zhejiang has also become an important concern because of its leading economy and education system in China. In this paper, we focus on in-service teachers’ STEM literacy improvement through the Zhejiang-Indiana Parallel Classroom Project (ZIPCP), which emphasises the introduction of international STEM teaching paradigms, teaching practices highlighting local culture and teachers’ reflections.

Methodology

Research questions

The empirical study was guided by the following primary research question: How does the ODR approach impact Chinese teachers’ professional development in STEM?

Professional development context

The context for professional development is ZIPCP, sponsored jointly by Zhejiang Department of Education and Indiana Department of Education. The objectives of ZIPCP are to improve teacher’s STEM literacy and to equip teachers with appropriate skills to design STEM education activities. Since STEM education has been practised in the United States for a long time, STEM courses, designed by teachers from Indiana teacher teams, were put into practice in Chinese schools in this project guided by their original designers. As STEM education has become an important topic in compulsory education in China, teachers from primary and secondary schools in Zhejiang province participated in the ZIPCP program to get support for their STEM teaching. With this approach, teachers have an opportunity to redesign STEM activities and to learn how to implement interdisciplinary activities in their own classrooms. Supervisors from government as well as professors from universities facilitated the sessions. The training materials and activities were developed collaboratively by teacher teams who had experience in STEM education and researchers who focused on STEM education.

Design and procedures

The study was divided into four stages: pre-test, teaching intervention, post-test and semi-structured interviews. A previous analysis was used to establish Zhejiang teachers’ STEM literacy (Huang, et al. in press). PD interventions with the process of observation, discussion and reflection were carried out through different training sessions. In this paper, we focus on the quantitative measures to investigate the effect of the intervention, but we are providing an overview of the entire data set to provide the context for the data and data analysis.

Selection of participants

The study focuses on 82 teachers from ZIPCP. Teachers come from primary and secondary schools, and most of them have some understanding of STEM education. These teachers had different academic backgrounds, including science, mathematics, technology and English. Twenty-eight teachers participated in the first week of training, named Group A. In addition, 8 teachers chosen from Group A participated in the training of the second week. Group B had 54 teachers who only participated in the second week. Table 1 shows the numbers and topics of different groups.

Table 1. Number of teachers and topics of different groups.

		Number
Variables	Category	Group A	Group B
Theme	Building a Car	7	13
	Buildings and bridges	7	13
	Aeronautical Engineering	7	14
	Series and parallel circuit	7	14

Teaching intervention

There were 4 topics of the STEM integration curriculum for every teacher to choose (see Table 1). Building a Car required students to use their knowledge of magnetic forces to create a car that moves through a student-created obstacle course. In Building a Bridge courses, students explored and manipulated shapes as they relate to natural and man-made structures. Aeronautical engineering required designing and building a tumble wing that travels the farthest distance and establishing a paper jet model. Series and parallel circuits required learning and creating a series or parallel circuit. All groups have experienced the process of observation, discussion and reflection, but the specific content and task requirements were different. (see Figure 1 and Figure 2)

Figure 1. Intervention between Group A and Group B.

Figure 2. The framework for observation, group discussion and reflection.

Group A: We divided its teaching process into 2 stages. Teachers were asked to experience a three-step cyclic process of lesson observation (20 hours), group discussion and video reflection in the first stage. They are asked to observe how American STEM teachers carry out teaching and record according to the scale. A 2-hour group discussion with no topic specified was arranged for the teachers to brainstorm and communicate after the observation. Reflect on American STEM teachers behaviour through video clips on the basis of classroom observation scale. The second stage is aloud reflection and report one topic related to STEM teaching. Group B: There were similar lesson observation, discussion and aloud reflection/report processes. The difference is that the task requirements are clarified, questions are clarified (8 hours) before the lesson observation (12 hours), and the observation and group discussion revolve around designated questions.

Instruments

The research tools used in this research are mainly the STEM Literacy Status Questionnaire, Interview Protocol and Observation Recording Table (see Table 2).

Table 2. Research instruments.

Type	Collection Means	Evaluation Tool
Quantitative data	Questionnaire	STEM Literacy Status Questionnaire (Pre- and post-test volume)
Qualitative data	Interview	Interview Outline of Teacher Interview (Groups A, B) Interview Transcription Text
	Class observation	Observation record table
	Discussion	Instructional design group discussion results

Questionnaires and interviews

The STEM Literacy Status Questionnaire (pre- and post-test versions) was compiled using STEM literacy related to scientific, mathematical, technology and engineering literacy assessment in The National Assessment of Educational Progress. The questionnaire is mainly used to examine the ability of teachers to comprehensively apply the four concepts of STEM to solve complex problems in real issues, the understanding and attitude towards STEM education. According to Zollman (2012), the ultimate goal of STEM education should change from learning STEM literacy to using STEM literacy for learning. Combined with Bloom educational goal taxonomy (cognitive domain (knowledge and process), emotional domain (attitude), psychological field), STEM literacy is divided into the following dimensions

• STEM knowledge (e.g. Scientific Knowledge. SK; Technical Knowledge, TK; Engineering Knowledge, EK; Mathematical Knowledge, MK; Integration Knowledge, IK)

• STEM ability (e.g. problem-solving ability, PSA; scientific inquiry ability, SIA; maths modelling ability, MMA; engineering design ability, EDS; technical application ability, TAA)

• STEM attitude (Attitudes towards connections between STEM disciplines, ATC; Attitudes towards effect of STEM education, AE; STEM career interest, CI)

Eighty-two valid questionnaires were used in this survey. After the questionnaires were collected, we evaluated and coded the teachers’ answers in the questionnaires, and SPSS was used for statistical analysis. The questionnaire used in this study included quiz and open questions. The quiz examines the two dimensions of teachers’ STEM concept understanding and STEM ability (see Table 3). The open questions are based on real-life situations, such as topics on wind generators, cross-sea bridges and air capture systems. The following example was using the topic of CO₂ Collector to investigate the level of SK (see Table 4). Different questions (including each sub-problem) have different aspects to examine. In the pre-test questionnaire, the first question mainly examines the teacher’s STEM understanding and ability. Considering that this question is biased towards the investigation of STEM teaching literacy, it contrasts with the second sub-question of the second question in the post-test questionnaire.

Table 3. Evaluation criteria for conventional questions.

Answer performance	Score
The teacher’s response is in complete agreement with the point of view of the questionnaire.	3
The teacher’s answer is partially consistent with the point of view that the questionnaire is to examine.	2
The teacher’s response is not related to the point of view of the questionnaire. The teacher’s answer is contrary to the point of view of the questionnaire.	1
Teacher did not answer	0

Table 4. Analysis of different levels of SK.

Level	Requirement	Example
Level 3	Choose the place where the carbon dioxide concentration is high (such as the factory, crowded places like the shopping mall, etc.); the place where the air flow rate is suitable; know the principle of the device usedand explain the reason accurately.	a)understand the principles of capturing CO2 and know how to deal with it b)choose a place with a large proportion of CO2 in space, such as the place with large population; c) the device can also be moved to collect CO2 more quickly, such as near a burning site; an external device such as an auricle that assists in collecting gas can be installed.
Level 2	The participants cannot clarify the reasons.	1.installed in the car exhaust where the concentration of CO2 is higher. 2.installed in the factory flue, the concentration of CO2 is higher, the effect is better.
Level 1	The reasons given are incorrect or not related to the key requirement.	Instal the system at the relevant place of the relevant enterprise and start the device at a fixed point and timing according to the production situation of the enterprise.

Data analysis

Data from multiple sources (pre-test, post-test and interview) were analysed. With regard to the status of teachers’ STEM understanding, we obtained data from participants (n = 82) in Zhejiang and used descriptive statistics to describe the significance of teachers’ STEM understanding and the aspects that should be emphasised. In addition to the analysis of teachers’ answers concerning the way of training to their STEM understanding, the significance of the teachers’ STEM understanding, including knowledge, ability and attitude, was analysed.

The scoring process was carried out by 6 STEM teachers with theoretical foundations and years of teaching experience, followed by consistency analysis. In order to analyse the quiz and the open questions, 6 STEM researchers with theoretical foundations and years of STEM teaching experience conducted a consensus evaluation according to the standards in Table 3 and Table 4. The Kendall’s coefficient of concordance reached 0.89 and 0.84, meaning that the results of evaluation were consistent. The consistency of each question is 0.8 and above, indicating that the text analysis has a relatively high consistency. The Likert scale of the questionnaire is a kind of scoring plus total scale with scientific and reasonable evaluation methods and is widely used in research evaluation. In this part, Level 5 is in favour, Level 4 is slightly in favour, Level 3 is general, Level 2 is a little disapproval and Level 1 is disapproval. Therefore, a higher average score in a category indicates that the dimension is well mastered.

Results and findings

The results indicate that teachers’ STEM literacy has improved, especially TK, EK and ATC, CI, and AE. Lesson observation, group discussion and reflection are effective approaches to improving teachers’ STEM literacy. In the following sections, we detail the evidence that supports such observations.

Comparative analysis of the STEM knowledge, ability and attitude dimensions

We tested the knowledge, ability and attitudes of different disciplines before and after the PD intervention, and observed the differences between the dimensions. It can be seen from Table 5 that the average value of SK (M = −0.163) decreased in the post-test compared with the pre-test, while the TK(M = 0.272), EK(M = 0.428), MK (M = 0.169), ATC(M = 0.339), CI(M = 0.700), AE(M = 0.998) increased and TK, EK, ATC, STEM CI, AE presented p < 0.001, were significant differences. The data show that there is no obvious effect on the understanding of teachers’ SK and PSA through training, but the understanding of MK, TK and EK, ATC, CI, AE has obvious effects.

Table 5. T-test in STEM knowledge and attitude dimensions.

Pre-Post Test (N = 82)
	M (SD)	T
Mathematical Knowledge (MK)	0.169 (0.754)	2.028*
Scientific Knowledge (SK)	−0.163 (0.741)	−2.002
Technical Knowledge (TK)	0.272 (0.692)	3.564**
Engineering Knowledge (EK)	0.428 (0.865)	4.482***
Problem-Solving Ability(PSA)	−0.131 (0.742)	−0.159
Attitudes Towards Connections (ATC)	0.339 (0.929)	3.306**
STEM Career Interest (CI)	0.700 (0.993)	6.373***
Attitudes Towards Effect of STEM Education (AE)	0.998 (2.059)	4.389***

Notes: *p < 0.05; **p < 0.01; ***p < 0.001

STEM discipline knowledge

We compared and analysed the understanding and application of teachers’ disciplinary knowledge in Group A and B. To comprehensively examine the data, we focused on a statistical analysis of the mastery of the concept of participating teachers from quantitative and qualitative analysis (see Table 6). The average value of Group A SK (M = −0.140) decreased post-test, while TK (M = 0.214), MK (M = 1.153), and EK (M = 0.321) increased. EK showed p < 0.05 and was significantly different, but SK, TK and MK were not significantly different. From the perspective of the distribution ratio of teachers at different levels, the proportion of the concept understanding of teachers in level 5 is mostly between 50% and 80%. People at level 5 of EK have increased from 50% to 75%.

Table 6. Comparison between Group A and B in STEM discipline knowledge.

	Group A (N = 28)		Group B (N = 54)		T(GroupA-B)
	M± SD	T(po-pr)	M+ SD	T(po-pr)
MKpre	4.329 ± .511	1.177	4.134 ± .623	1.643	1.419
MKpost	4.482 ± .535		4.311 ± .513		1.410
SKpre	4.655 ± .532	−0.954	4.580 ± .395	−1.777	0.721
SKpost	4.515 ± .503		4.404 ± .639		0.797
TKpre	4.406 ± .409	1.921	4.050 ± .464	2.992*	3.431**
TKpost	4.620 ± .442		4.398 ± .615		\
EKpre	4.161 ± .559	2.588*	4.007 ± .659	3.716**	1.054
EKpost	4.482 ± .486		4.491 ± .608		−0.065

Notes:Pre:Pres-test; Post: Post-test.

The average value of Group B in SK (M = −0.176) decreased, while TK (M = 0.348), EK (M = 0.484), and MK (M = 0.177) increased. However, EK showed p < 0.01, and TK showed p < 0.05. The results showed that after training, the EK and TK of group B was significantly improved, while the levels of MK and SK were not significantly different from those of group A. The proportion of teachers who had a basic understanding of TK rose from 40.74% to 62.96%. Moreover, EK improved significantly, and the proportion increased 29.63%(see Table 7).

Table 7. Analysis of STEM discipline knowledge understanding.

Levels		SK		TK		EK		MK
		Pre	Post	Pre	Post	Pre	Post	Pre	Post
A	Level 5	82.1%	82.1%	75.0%	85.7%	50.0%	75.0%	67.9%	71.4%
	Level 4	14.3%	14.3%	25.0%	14.3%	46.4%	25.0%	28.6%	28.6%
	Level 3	3.6%	3.6%	0.0%	0.0%	3.6%	0.0%	3.57%	0.0%
B	Level 5	83.3%	70.4%	40.8%	63.0%	44.4%	74.1%	51.9%	75.9%
	Level 4	16.7%	24.1%	53.7%	31.5%	44.4%	22.2%	44.4%	20.4%
	Level 3	0.0%	5.6%	5.6%	5.6%	9.3%	3.7%	1.9%	3.7%
	Level 2	0.0%	0.0%	0.0%	0.0%	1.85%	0.0%	1.8%	0.0%

There were no members in Group A reaching level 1,2 and Group B in level 1

For IK, most teachers lacked an understanding of interdisciplinary concepts, and few teachers had a good understanding of interdisciplinary concepts . Most of them had deviations from expected outcomes. Most teachers had a vague understanding of subject knowledge. They simply consider STEM integration as a simple superposition of multiple disciplines. For example, a teacher with a mathematical background knows the engineering discipline contains measurements. In the teaching design process, he expressed that he knows engineering and mathematics should be closely integrated, but he did not know how to integrate them. Only a small number of teachers can use the knowledge of various disciplines to solve practical problems, thus achieving cross-disciplinary integration.

Overall, the findings are as follows: (a) both groups have improved in TK and EK, MK; (b) Group B improved more than Group A in EK and TK; (c) These teachers still performed at a lower level of IK which to be the most pressing aspect that need further PD.

STEM ability

Problem-solving ability (PSA)

Comparing the pre- and post-test the average value of the problem-solving ability in Group A increased, but group B decreased, and group A (M = 0.098) was improving more than group B (M = −0.071), and p < 0.05, there was significant a difference between Group A and B (see Table 8). Furthermore, the percentage chart shows that most teachers in Group A are at level 5 and level 4, and more than 80% of teachers have basic problem-solving ability. In the open question, most teachers can provide solutions to problems based on the comprehensive use of multidisciplinary knowledge and can optimise existing solutions and exchange design concepts. For example, Chen CY (pseudonym) designed bridges. The 40 m bridge has a moderate span and does not require a cable-stayed structure, but the span is too large for a single-hole arch bridge, so she combined the advantages of both to design. However, the ability of teachers in Group B declined after training; the proportion of teachers at level 5 dropped from 74.07% to 62.69%, but other levels improved (see Table 9). Through the comparative analysis, Group B decreased, but the problem-solving ability of Group A improved. Among the teachers in each group, teachers with basic problem-solving ability accounted for more than 70% and performed well. However, there are still some problems. Most teachers still lack awareness of comprehensively considering the complexity of the problem.

Table 8. Comparison between A and B in STEM pedagogical skills.

	Group A (N = 28)		Group B (N = 54)		T (GroupA-B)
	M ± SD	T(po-pr)	M ± SD	T(po-pr)
PSA pre	4.438 ± .331	0.990	4.330 ± .383	−0.621	1.273
PSA post	4.536 ± .415		4.259 ± .716		2.211*

Table 9. Analysis of pedagogical skills in STEM.

Problem-solving Ability (PSA)
	Group A		Group B
	Pre-test	Post-test	Pre-test	Post-test
Level 5	85.71%	78.57%	74.07%	62.69%
Level 4	14.29%	21.43%	25.93%	20.90%
Level 3	0.00%	0.00%	0.00%	7.46%

Table 10. Comparison between A and B in STEM attitude.

	Group A (N = 28)		Group B (N = 54)		T(Group A-B)
	M ± SD	T(po-pr)	M ± SD	T(po-pr)
ATCpre	4.051 ± .633	3.729**	4.211 ± .702	1.714	−1.010
ATCpost	4.600 ± .478		4.441 ± .547		1.303
CIpre	3.953 ± .554	3.719**	3.604 ± .623	5.212***	2.494*
CIpost	4.536 ± .782		4.365 ± .767		0.953
AEpre	3.756 ± .815	4.275***	3.759 ± .708	6.088***	−0.021
AEpost	4.583 ±.495		4.525 ± .544		0.565

Overall, a small number of teachers have strong problem-solving ability and a good grasp of the problem-solving process, including describing and designing problem solutions; some teachers’ understandings are incomplete and often involve only part of the process. A small number of teachers have clear problem-solving awareness, which includes processing the information in the material to further clarify the problem, using mathematical modelling and scientific inquiry to design the problem solution, and reflecting on the problems existing to improve. Teacher Xie CY’s systematic approach used is to a)find problems b)finding constraints c)brainstorming d)choice options e)build prototypes f)testing g)repetitive testing h)communication and discussions. The teacher has a good grasp of the entire problem-solving process.

Scientific inquiry ability (SIA)

Teachers, especially those with scientific backgrounds, can reflect a certain precognition and knowledge reserve for scientific inquiry ability(SIA) in the pre-test. From the process data, it can be seen that most teachers do not have enough understanding of SIA. In addition, there is confusion about its use in teaching. A small number of teachers misunderstood ‘scientific inquiry’ and considered it an ‘engineering design process’ or ‘reading material practice’. Some teachers chose a scientific question as a question of inquiry, such as ‘the production of rockets’ and ‘the research of proportion’, suggesting certainty for scientific inquiry of understanding and application. However, it is not enough for the understanding and application of setting up questions in scientific inquiries. Except for some teachers who have a high level of understanding in the background of science education, most teachers seem to have fairly limited understanding. In SIA, there is little difference between the two groups of teachers, which is greatly influenced by academic background.

STEM attitude

Compared with the pre-test, the mean values of ATC (M = 0.549) and CI (M = 0.583) and AE (M = 0.827) in group A increased, and P < 0.05 showed a significant difference (see Table 10). The mean values of CI (M = 0.761) and AE (M = 0.766) in group B increased and P < 0.001, it showed significant difference, but ATC in Group B had no significant difference. The understanding has been significantly improved in the STEM attitude. On the whole, for the relationship between learning mathematics and STEM education, the teacher who believes that ‘learning mathematics helps us to learn relevant content in engineering, technology, science and other fields’ is approximately 80% in the pre-test and approximately 72% in the post. Teachers generally understand the relationship between learning mathematics and STEM education. There is little difference between the two groups (Table 10).

Before the intervention, most teachers agreed that the teaching of each subject should strengthen the connection with society and life. For the overall understanding of STEM, teachers had a vague cognition of ‘STEM is a course’ or ‘STEM is a teaching concept’. After training, the definition of STEM and its characteristics were better understood. On the whole, the understanding of STEM has been improved through this activity.

Conclusions and discussions

Given the challenges faced by many teachers worldwide, many teachers are lack of STEM education experience (Awad and Barak 2018; Honey, et al. 2014). The present study contributes to understanding how teachers’ STEM knowledge, PCK in STEM knowledge and attitudes may be improved. Teachers should be given STEM learning experiences and be encouraged to carry out STEM curricula by providing real situation questions to deepen the understanding of STEM concepts and pedagogical strategies that support STEM teaching (e.g. problem-solving ability, scientific-inquiry ability, critical thinking) (Bybee 2013; Margot and Kettler 2019; Zollman 2012). The PD programmes prompted teachers to explore the STEM class in depth, deepen or correct the teachers’ original understanding of STEM and encouraged teachers to understand the STEM curriculum. Many studies investigate how STEM PD program impact in-service teachers’ conceptualisations and instruction of STEM (Akerson and Buck 2020; Berry, et al. 2019), fewer still have discussed this impact teachers’ STEM literacy through observation, discussion and reflection approach.

Overall, the observation, discussion and reflection approach was effective for teachers’ PD in STEM. Through this PD training, teachers’ understanding of EK, TK, MK; PSA, ATC, CI and AE improved. Lesson Observation and Group Discussion encourages teachers to learn STEM knowledge and pedagogical change in a timely mannerand be deepened by discussion. Both groups have experienced Lesson Observation and Group Discussion, so these may have a positive effect on the improvement of STEM literacy. The results of this study indicate that the ODR approach has the potential to improve teachers’ understanding and attitudes towards STEM teaching. Observation encourages teachers to understand STEM classrooms and seeks the use and integration of engineering design, scientific inquiry, mathematical modelling and technology applications from teaching (Shernoff et al. 2017). The discussion gives teachers the opportunity to communicate with each other, deepen the connection between STEM, society and life, and learn to solve problems in real situations. Video Reflection is an effective way for teachers’ reflection to improve their in-depth thinking and problem-solving ability. This STEM PD training lasted one week and had different approaches. Both Group A and B have grown in different aspects. The teachers who participated in the two-week training have significantly improved their knowledge, abilities and attitudes in STEM and have turned the STEM education they have learned into STEM teaching and have begun to shift from STEM literacy to teachers’ PCK for STEM. Teachers value PD when they can acquire the knowledge and skills in order to improve their students’ STEM literacy, for example by improving their PCK (Hwang, et al. 2018). Aspects of teachers’ knowledge such as PCK have not been investigated in the present study. Future studies could explore how the ODR framework may support teachers’ PCK for teaching STEM.

Acknowledgments

We would like to thank the the National Social Science Fund of China for its support (BHA210121), Department of Education of Zhejiang Province, Teaching and Research Section of Zhejiang, who supported the training project of ZIPCP; the teachers and students involved in the research;, and the anonymous referees who provided us with constructive comments and suggestions.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

The research was supported by the National Social Science Fund of China (Grant number: BHA210121).