When engineering design meets STEAM education in hybrid learning environment: teachers’ innovation key through design heuristics

ABSTRACT

Implementing STEAM education in classrooms can be enhanced by incorporating other methods to bring students flexibility during learning, such as the engineering design process (EDP) and hybrid learning. We developed design heuristics to support teachers in enhancing the quality of STEAM lessons, emphasizing EDP within hybrid learning to assist teachers in developing lesson plans. The design heuristics were implemented in three groups of online professional development programs within educational design research, with a hundred and eighty-three teachers submitting their lesson plans. We scored the lesson plans and conducted the Kruskal Wallis H test procedure. The result showed that teachers who participated in this study were likely to incorporate the key principles of design heuristics in their lesson planning practices. However, we found that there is a necessity for further exploration, especially in the variation of assessment strategies and the exploration of problems or context for STEAM learning. Therefore, the result emphasized the significance of future exploration that defines and formulates problems for STEAM activities and its implications on teachers’ lesson planning. This study harmonizes technological advancement and pedagogical practices within hybrid learning by integrating emerging technologies into lesson planning and evolving a design approach into the STEAM curriculum.

KEYWORDS:

• Design heuristics
• STEAM education
• engineering design process
• hybrid learning
• pedagogical guidelines
• teacher professional development

Introduction

Design heuristics are guidelines for generating ideas that can be identified as a protocol for performing design tasks (Yilmaz et al., 2012). Furthermore, the importance of design heuristics has been reported, particularly the way in which it supports idea generation in an educational context. For example, a study carried out by Schallert et al. (2022) reported that the development of design heuristics based on the 5E inquiry model was reported to support teachers’ lesson-planning activities in the flipped classroom adequately. Moreover, two studies also underlined the importance of guided reinvention to improve creativity through design heuristics (Gravemeijer & Bakker, 2006; Yilmaz et al., 2012). Furthermore, Gravemeijer and Bakker () argue that the impact of design heuristics can contribute to developing a good framework in statistics education, which could empower educational processes. Therefore, employing design heuristics as pedagogical support for teachers in developing lessons is essential.

Numerous studies highlighted the importance of pedagogical support for teachers in implementing STEAM (Science, Technology, Engineering, Arts and Mathematics) lessons, hybrid learning, and the engineering design process (EDP). For example, Diego-Mantecón et al. (2022) stated that teachers’ competencies and understanding could influence students’ learning in exploring EDP in STEAM activities. Moreover, Belbase et al. (2021) reported that there is a need for proper guidance to integrate STEAM education effectively to support teachers’ lesson preparation. Additionally, a systematic literature review conducted by Raes et al. (2020) revealed the importance of design guidelines from pedagogical perspectives for setting up synchronous hybrid learning. Referring to this, it seemed evident that the importance of design heuristics could benefit this current study and future studies, particularly in supporting the development of STEAM education.

Despite the potential of EDP and hybrid learning for STEAM learning to provide teachers with opportunities to facilitate students’ learning, a question remains about how to design STEAM learning based on EDP in hybrid learning scenarios. Hence, the novelty of this study lies in manifold aspects related to design heuristics as pedagogical guidelines for teachers. Firstly, we found a valuable opportunity to implement STEAM lessons with EDP (L.D. English & King, 2019; Fidai et al., 2020) by looking for opportunities to employ hybrid learning, which appears suitable for facilitating STEAM activities (Boon, 2021; Borba, 2021; Linder, 2017; Moskal, 2017). Secondly, Mishra et al. (2020), Butz and Stupnisky (2016), and Raes et al. (2020) revealed the need for guidance for teachers to deal with challenges in implementing hybrid learning. Thirdly, we continued the work of several studies that have reported the use of design heuristics to provide teachers with proper pedagogical guidance in developing lesson plans (e.g., Davis & Krajcik, 2004; Janssen et al., 2009; Schallert et al., 2022). Fourthly, Schallert et al. (2022) stated that design heuristics might offer opportunities to support teachers’ lesson-planning skills. The earlier research motivated us to concentrate on designing heuristics that could help guide teachers in designing STEAM lessons focusing on EDP in hybrid learning.

Furthermore, the following overarching research questions (RQ1 and RQ2) guided this study to investigate how well the development of iterative design heuristics was helpful in developing lesson plans:

1. RQ 1: What is the difference between teachers’ scores based on EDP factors of their lesson plans regarding all groups?

2. RQ 2: What is the difference between each factor of EDP regarding groups based on the analysis of lesson plans regarding all groups?

In a nutshell, this manuscript, due to size limitations, only reports the findings from one of the three data sources that were obtained. The results reported here are also largely supported by the other two data sources and will be reported in future papers. Subsequently, we organized this paper into the following sections: literature reviews, methodology, findings, and implications. In the subsequent section, we first describe perspectives of hybrid learning and engineering design process (EDP) in STEAM education. The next section provides our methods employed and discusses the main research stages. Finally, we discuss the results and implications derived from the research findings.

Literature reviews

With respect to the main objectives of the study, it is important to understand how well the development of iterative design heuristics is helpful in developing lesson plans. The following literature reviews employ the outlooks of hybrid learning and STEAM education. Furthermore, we provided current research related to the concept of Engineering Design Process (EDP) in STEAM education to underline the importance of EDP in supporting teachers to develop lesson plans as well as the development of the instrument used in the methodology employed in this study.

Outlooks of hybrid learning in classroom settings for STEAM lessons

Hybrid learning can be defined as an intermediary stage between fully classroom-based and entirely virtual instruction. Hybrid learning combines face-to-face and online instruction with technical assistance (Klimova & Kacetl, 2015). Aligned with the previous argument, some researchers also emphasize using technology to support hybrid learning. Borba (2021), Linder (2017) and Moskal (2017) stated that combining in-person and technology-mediated learning is called hybrid learning. Hence, hybrid learning is a learning environment that is different from traditional face-to-face activities that involve technology-enhanced activities. Thus, it can be understood that hybrid learning may enable diverse students with different learning styles, backgrounds, needs, and geographic locations to enact their learning activities.

Numerous studies have documented that hybrid learning has the potential to bring many benefits in facilitating students learning. Moskal (2017) and Rodriguez-Paz et al. (2021) in their research reported that hybrid learning could improve students’ learning quality and motivation. Linder (2017) argued that hybrid learning might support students’ engagement and interaction with teachers and peers in various ways as it provides opportunities for students to join in person or online. This argument is supported by some researchers that urged the implementation of hybrid learning, which offers students various learning experiences and flexibility (Borba, 2021; Linder, 2017; Moskal, 2017; O’Byrne & Pytash, 2015). Therefore, hybrid learning has been shown to yield various advantages in promoting STEAM learning.

Technological development has recently been rapidly increasing, allowing teachers to conduct face-to-face and online learning in real-time with hybrid learning (Laksmiwati et al., 2021; Mishra et al., 2020; Singh et al., 2021). For instance, teachers can conduct modified learning by utilizing technology-enhanced activities. This method allows online and in-person students to participate in real-time learning activities, called synchronous hybrid learning (Butz & Stupnisky, 2016). As Boelens et al. (2017) pointed out, teachers can facilitate learning that addresses their needs by enabling students to engage in classroom activities through synchronous hybrid learning. Therefore, it should be noted that this study explores how STEAM lessons focusing on EDP can be implemented in a hybrid learning environment.

Furthermore, numerous research has reported that design heuristics are remarkably grounded in supporting teachers in facing challenges in developing processes and implementation. For example, Leahy et al. (2018) revealed that design heuristics could be applied to generate and help to develop initial concepts. The prevalence of design heuristics to facilitate individual idea generations in eliciting creative concepts as guidelines have been reported to better influence students’ conceptions (Yilmaz et al., 2012). Moreover, some researchers argue that design heuristics can support idea generation and lesson-planning activities (Davis & Krajcik, 2004; Janssen et al., 2009; Schallert et al., 2022). Therefore, in this study, the design heuristics are exploited as cognitive approaches that can be applied as strategies to help teachers develop STEAM hybrid lesson plans.

Perspectives of engineering design process for STEAM activities

Empirical research conducted in recent years has reported the importance of design thinking activities in interdisciplinary learning. For instance, Li et al. (2019) documented the importance of the design process in science, technology, engineering, and mathematics activities. Additionally, L.D. English and King (2019) and Atman et al. (2007) revealed that the engineering design process (EDP) plays a crucial role in teaching and learning practices. Moreover, a study by Sullivan and Bers (2018) reported the opportunities for integrating EDP in STEAM activities through robotics. Another study conducted by Khamhaengpol et al. (2021) also reported a positive impact on integrating EDP in STEAM learning to students’ learning outcomes. Thus, the studies mentioned earlier underlie our interest in investigating EDP in STEAM lessons in this study context.

Moreover, EDP has been demonstrated to develop students’ thinking and problem-solving skills. Some argue that EDP provides opportunities for students to develop problem-solving, critical thinking, communication, self-confidence, and other 21st-century skills (Stohlmann et al., 2014; Syukri et al., 2018). In addition, EDP emphasizes systematic problem-solving and using open-ended ways to design solutions and aims to encourage students’ critical and creative thinking through trial-and-error activities (L.D. English & King, 2015; Hafiz & Ayop, 2019; Jolly, 2017; Li et al., 2019). Some studies emphasize that in EDP, students explore through some phases that aim to facilitate students conducting investigations (L.D. English & King, 2015; Jolly, 2017; Li et al., 2019). Furthermore, some researchers have underlined that EDP includes research or investigation activities that begin with identifying open-ended problems and end with ideal solutions (Haik & Shahin, 2011; Lockledge & Salustri, 1999).

Based on the discussion mentioned earlier, EDP can be conceptualized as a process emphasizing systematic problem-solving and using an open-ended design method. EDP was built as an iterative approach, offering a sequence of practices that form an iterative process consisting of (1) defining the problem, (2) idea generation, (3) designing and constructing, (4) design evaluation and redesign, (5) evaluation and reflection (L.D. English, 2016; L.D. English & King, 2019; Han & Shim, 2019; Jolly, 2017; Simarro & Couso, 2021). EDP provides opportunities for students to conduct experiments and is utilized in this study as the primary learning activity of STEAM lessons developed by teachers. In further depth, we illustrate the STEAM activities that focus on EDP in the following descriptions.

1. Defining the problem: The teacher introduces a contextual problem relevant to STEAM education for students to explore. The problem aims to encourage students to investigate and promote their curiosity. The classroom discussions include the needs and limitations of solving the problem and the success criteria.

2. Idea generation: This stage enables students to engage in discussions and brainstorm potential solutions or prototypes for the problem presented by the teacher. Moreover, the teacher encourages students to share their ideas, develop strategies, and propose multiple solutions related to STEAM activities. In their group, students continue the discussion by sharing and developing strategies and simulations in their groups, then selecting the best possible solutions to be executed in the subsequent exploration of STEAM activities.

3. Designing and constructing: This stage expects the teacher to emphasize that students actively participate in practical exercises. Moreover, the teacher encourages students to model the best possible solution(s) by designing and constructing the solution(s) from their previous stage. The teacher plays a vital role in providing opportunities for students to address the problem in various innovative ways.

4. Design evaluation and redesign: The main ideas of this phase consist of identifying solutions, redesigning and selecting the ideal solutions, and communicating the design process. This stage aims to challenge students to test and assess the model as part of the trial-and-error activities. The teacher encourages students to revise their prototypes in light of evaluation.

5. Evaluation and reflection: The main activities of this phase are evaluating design process activities and communicating their investigations. In this stage, the teacher needs to provide concise instructions that encourage students to reflect on the significance of each phase. Therefore, students reflect on their design process during classroom discussions and communicate their reflection results.

Methodology

We organized the methodology into several sections, starting with formulating our study context by corroborating the literature review as the significant reference of our study. Moreover, we present the research hypothesis, continuing with information on the participants and the implementation process. In the final section, we describe the way in which the data was collected and analyzed during the study activities.

Study context

We employed the perspectives of EDP for STEAM activities in the theoretical framework, especially in establishing the instruments and the design heuristics. Furthermore, the theoretical framework offers a comprehensive understanding of enhancing the lesson planning activities through design heuristics by understanding how the views of EDP in STEAM lessons and hybrid learning underpin this study. In addition, the concept of EDP influences our methods in developing instruments and conducting analysis of teachers’ lesson plans. Given the significance of previously stated empirical studies as essential for teachers’ guidance, as previously mentioned in the introduction and literature review sections, we investigated pedagogical supports necessary for teachers’ lesson planning activities (the design heuristics in Figure A1). In this current investigation, we present our investigation on how design heuristics can potentially expedite teachers in developing STEAM lesson plans based on EDP in hybrid learning. Hence, this manuscript outlines the utilization of design research activities to guide teachers in developing STEAM lessons as part of our design research project.

Research hypotheses

This study is driven by a set of research hypotheses, leading to a broader understanding that is presented in the findings to address the research questions. We formulate the following hypotheses to help in the contributions of our research questions:

(1) Related to the first research question (RQ1), we formulate the following hypothesis: Because of the development of design heuristics and implementations, we hypothesize (H01) that “There is no statistically significant difference between teachers’ scores based on EDP factors of their STEAM-EDP hybrid learning lesson plans regarding all groups”.

• H01a: There is no statistically significant difference between teachers’ scores based on EDP factors regarding groups 1 and 2.

• H01b: There is no statistically significant difference between teachers’ scores based on EDP factors regarding groups 1 and 3.

• H01c: There is no statistically significant difference between teachers’ scores based on EDP factors regarding groups 2 and 3.

(2) Related to the second research question (RQ2), we formulate the following hypothesis:Because of the development of design heuristics and implementations, we hypothesize (H02) that “There is no statistically significant difference between each factor of EDP regarding groups based on the analysis of STEAM-EDP hybrid learning lesson plans”.

• H02a: There is no statistically significant difference between each factor of EDP regarding groups 1 and 2.

• H02b: There is no statistically significant difference between each factor of EDP regarding groups 1 and 3.

• H02c: There is no statistically significant difference between each factor of EDP regarding groups 2 and 3.

Participants

In total, 190 teachers participated in this study, with 183 teachers submitting their lesson plans. Fifty-five, forty-nine, and seventy-nine teachers submitted lesson plans for the first, second, and third groups, as summarized by the participants in Table 1. All participants in each group were different. Therefore, there was no overlap ratio among teachers who submitted the lesson plans.

Table 1. Study group.

Category		Group
		1	2	3
Gender	Female	33	38	51
	Male	24	16	28
Age	<25 years old	19	2	0
	25–35 years old	21	11	16
	36–45 years old	14	33	44
	46–55 years old	3	6	18
	>55 years old	48	2	1
School funding	Governmental	31	43	68
	Non-Governmental	9	11	11
School level	Elementary School	8	26	12
	Junior High School	25	16	32
	Senior High School	24	12	35
Educational background	Bachelor	26	34	52
	Master	27	20	27
Working experiences	Short (<5 years)	28	4	4
	Medium (5–15 years)	23	31	37
	Long (>15 years)	33	19	38
Total		57	54	79

The sampling procedures were a combination of purposive and voluntary sampling. We collaborated with four Indonesian teacher communities, and participating teachers willingly expressed their interest in joining our study without coercion. Participating teachers in the online workshop took part in our study to investigate the contribution of design heuristics to teachers’ lesson-planning activities. The workshops were aimed at facilitating teachers’ learning of how to utilize design heuristics in their lesson-planning activities. Following the workshops, each participant designed a lesson plan, which was used to gather evidence for the study’s goal. Therefore, all participants have become acquainted with the concepts of STEAM education, EDP, and hybrid learning. However, the workshops were conducted online, and we have no information about their participation in other professional development programs in the meantime. Thus, we consider this condition as uncontrolled extraneous, which makes it hard to analyze the influence of this study.

Research design and implementations

This study was part of an educational design research project of the first author’s doctoral study in facilitating STEAM learning in Indonesia. The design research activities involved the initial, development, and evaluation stages (Bakker, 2018; McKenney & Reeves, 2020). The activities of each stage are as follows.

1. The initial stage of the project focuses on the preparation and design, including agenda setting and diagnosing, developing conceptual framework, and exploratory activity of Indonesian STEAM education implementation. In this study, the practical issue was derived into the main research questions and the formulation of a conceptual framework.

2. In the development stage, the main activities were implementation and analysis, employing collaborative design with teachers through online workshops and classroom implementations to gather empirical findings by conducting iterative cycles.

3. The last stage, evaluation, focuses on the refinement of the pedagogical guidelines, including evaluation and reflection.

We utilized the design heuristics to assist teachers in developing STEAM lesson plans, emphasizing EDP as part of the research implementations. The activities aimed to learn from the utilization of pedagogical guidelines in supporting teachers in developing lessons. In the process of lesson plan development, teachers were given design heuristics and examples of the lesson plans to assist them. The design heuristics used in every group were presented as guidelines with several key principles for teachers in developing STEAM lesson plans focusing on EDP in hybrid learning. Consequently, in this study, we examined every group by comparing the scores of each group. This paper presents our investigation of the contribution of design heuristics in each group of implementations by examining three distinct groups from three different implementations within design research activities, as explained in the following section.

Data collection and analysis

Three implementations through online workshops by three different groups of teachers were conducted as part of the investigation. The online workshops were conducted three times with the three groups of teachers, with 32 hours of learning in total, divided by 22 hours for developing lesson plans, 6 hours for the rationale of STEAM education and hybrid learning and 4 hours for discussion focusing on course evaluation and feedback. The workshops were conducted in August-September 2022, September-October 2022, and January-February 2023 for the first, second, and third workshops, respectively. In the online workshops, we introduced STEAM education, the concept of EDP and hybrid learning. We facilitated discussion among participants by providing a breakout room. We introduced how to utilize GeoGebra online activities and classrooms, particularly to take advantage of GeoGebra resources. Meanwhile, we provided teachers with guidelines to support their lesson-planning activities. Teachers were asked to develop lesson plans as their tasks and submit them at the end of the course. The following Figure 1 summarizes the activities during the online workshop.

Figure 1. The structure of the program of online workshops.

We conducted the data analysis to provide empirical evidence on how the development design heuristics might give different contributions to assisting teachers with lesson-planning activities based on groups. During the workshop, every teacher was given a task to submit a lesson plan, which was subsequently reviewed and analyzed as the primary source of data for this study. We analyzed 183 lesson plans, as reported in Table 2. The p-value of all categories based on the Shapiro-Wilk test demonstrated that the data were not normally distributed as the p-value < .001. Therefore, the Non-Parametric Kruskal-Wallis’s test procedure was followed in the data analysis. We used Jamovi 2.3.21 software to analyze data by conducting a non-parametric Kruskal-Wallis test at a p ≤ 0.05 significance level. The Cronbach’s Alpha to measure internal consistency and reliability for all constructs from three groups was 0.978, indicating high reliability and internal consistency. Overall, the measurement strengthens the trust in the validity and robustness of the research findings.

Table 2. Descriptive statistics of individual scores and EDP factors.

	Individual Scores	EDP Factors
		LC	DP	IG	DC	DR	ER	AC
N	183	183	183	183	183	183	183	183
Mean	98.7	24.2	8.96	12.4	12.1	15.2	11.2	12.3
Median	101	25	10	12	12	16	12	12
Standard deviations	19.6	4.24	1.46	2.81	3.01	4.20	3.05	2.12
Skewness	−0.404	−0.672	−1.32	−0.866	−0.760	−0.466	−0.331	−0.486
Std. error skewness	0.180	0.180	0.180	0.180	0.180	0.180	0.180	0.180
Kurtosis	−0.838	−0.0572	1.14	−0.0421	−0.269	−0.630	−0.735	−0.405
Std. error kurtosis	0.357	0.357	0.357	0.357	0.357	0.357	0.357	0.357
Shapiro-Wilk W	0.944	0.943	0.717	0.826	0.849	0.885	0.895	0.932
Shapiro-Wilk p	<.001	<.001	<.001	<.001	<.001	<.001	<.001	<.001

Instrument

A questionnaire entitled Engineering Design Process Rubric Lesson Plans (EDPLPR) was utilized as the instrument of this study. The questionnaire consisted of items to evaluate the quality of teachers’ lesson plans developed in this study. To contribute to the research questions, the EDPLPR is intended to evaluate teachers’ STEAM lesson plans based on EDP. There are 24 items in the rubric related to STEAM lesson guiding principles, learning cycle, and lesson plan components with Likert-scale type 5 (1 to 5), as can be indicated in Table A4 in the Appendices.

Furthermore, a psychometric development was carried out to examine the instrument’s structure. The psychometric development aims to examine the quality of items, whether they are aligned with the EDP, and whether the scoring criteria are clear and understandable. Five researchers who have had experiences with teaching, EDP, and STEAM activities for over four years have contributed to aiding the content quality of EDPLPR. The following analyses were carried out as a confirmatory factor analysis to establish evidence of construct validity. Based on the confirmatory factor analysis, the EDPLPR had a reasonable fit based on criteria CFI-value = 0.940 and TLI-value = 0.928 > 0.90, which indicates an acceptable fit (Schumacker & Lomax, 2015). Moreover, the value of SRMR = 0.038 ≤ 0.05 and RMSEA = 0.079 < 0.08 indicates the instrument’s satisfactory fit (Schumacker & Lomax, 2015). Moreover, the value of the chi-square analysis χ² = 545 (df = 231), resulting in a ratio of χ² and df is less than 3 (χ²/df = 2.35 ≤ 3), which satisfied the general rule for acceptable model fit (Schreiber et al., 2006; Schumacker & Lomax, 2015). In addition, the EDPLPR instrument satisfied the construct validity with all factors exceeding 0.6, which fulfil high-reliability criteria, as reported in Table 3. The analysis demonstrated that the EDPLPR instrument is valid and reliable for measuring STEAM lesson plans focusing on EDP. Hence, based on the factor analysis, the EDPLPR satisfied an acceptable model fit to evaluate teachers’ lesson plans based on EDP.

Table 3. Construct validity test.

Factor	Cronbach Alpha	Construct Reliability
Guiding principles	0.871	0.717
Defining the problem	0.856	0.922
Idea generations	0.914	0.940
Designing and constructing	0.919	0.947
Design evaluation and redesign	0.964	0.981
Evaluation and Reflection	0.960	0.986
Additional Components	0.718	0.641

Interrater reliability

In this activity, the first author worked with another researcher with EDP experience to establish interrater reliability. We conducted two meetings to discuss the analysis process and build the same understanding. In the first meeting, we discussed the analysis procedure and criteria to reach a consensus on the EDPLPR rubric’s criteria and scoring procedures. After the first meeting, we worked independently to score the same twenty lesson plans. Afterwards, we discussed the analysis activities for the second time and conducted an interrater reliability analysis. As a result, the analysis of Cohen’s Kappa was used to evaluate the agreement, indicating substantial agreement between the raters (κ = 0.784). After reaching a consensus, the first author continued working independently on the data analysis.

Findings

We present the result in the following section, which relates to how the heuristics can support teachers in developing STEAM lessons based on EDP in hybrid learning based on the main research questions. In the first section, we provide the findings related to the first research question (RQ1) and report the data analysis results of teachers’ scores. The subsequent results obtained the second research question (RQ2) relating to EDP factors across all groups conducted in the three online workshops.

Results related to the first research question (RQ1)

We summarized the non-parametric test procedure for individual scores in Tables 4 and 5. Table 4 reported significant differences between individual scores with large effect sizes (ε² = 0.336). Furthermore, for the median of the individual score, we found that the median and mean scores were 101 and 98.7, respectively. Based on the data analysis, there is a significant difference between individual scores, as presented in Table 4 (p < .001). Thus, it can be concluded that there is a significant difference between teachers’ scores based on EDP factors of their STEAM-EDP hybrid learning lesson plans regarding three groups (H0 was rejected).

Table 4. Individual scores with non-parametric Kruskal-Wallis test.

Category	Mean (SD)	Median	χ^2	df	p	ε^2
Individual Score	98.7(19.6)	101	61.2	2	<.001	0.336

Significance at p < 0.05.

Table 5. Dwass-Steel-Critchlow-Fligner pairwise comparisons for individual scores.

Group		W	p
1	2	0.415	0.954
1	3	9.665	<.001
2	3	8.763	<.001

Significance at p < 0.05.

Meanwhile, further investigations, as reported in Table 5, showed that the result of pairwise comparison depicted no significant difference between Groups 1 and 2. Although there is a significant difference for all groups, Table 5 shows that the significant difference only indicates Groups 1 and 3 (H01b was rejected) and Groups 2 and 3 (H01c was rejected). Hence, it can be indicated that there is no significant difference between Groups 1 and 2, as we accepted H01a. To support these findings, we found that there are increasing mean scores from Group 1 (M1(SD) = 84.98 (14.09)), Group 2 (M2(SD) = 85.24 (17.97)), and Group 3 (M3(SD) = 106.20 (16.19)) (please see Table A1 in the Appendices). Similarly, as summarized in Table A2 (Appendices), if we refer to the median of the individual scores, they are slightly different for Groups 1 and 2, with 2 points increased (from 89 to 91). However, there were 27 increasing points for groups 2 and 3 (from 91 to 118, as reported in Table A2).

Results related to the second research question (RQ2)

Based on the data analysis, as summarized in Table A1, the average score of EDP factors for the first group was 3.69 out of 5 (73.4 %). This score slightly rose to 3.70 out of 5 (74%) in the second group and further climbed to 4.42 out of 5 (88.4%) in the third group. The findings showed that there was a minimal increase in the average score when comparing the first and second groups, with only around 0.01 or 0.2% (as reported in Table A1). In contrast, the increasing score from the second to third group was significantly higher, with a 0.72 or 14.4% difference in the mean score. Thus, despite the relatively minimal increase in average scores between groups 1 to 2 was small compared to the increasing score when comparing groups 2 and 3, the findings nonetheless indicated that the quality of teachers’ lesson plans accomplished remarkable performance.

The findings demonstrated that there were positive contributions of design heuristics for teachers in setting up lessons, especially after the third implementation. For instance, based on the mean scores and the median (as reported in Tables A1 and table A2 in the Appendices), it can be observed that there is an increasing number of teachers’ mean and median values. The highest scores for both mean and median were achieved after the third implementation. Furthermore, based on the analysis result for item 5 (LC 5), it can be observed that item 5 (LC 5) of Factor 1 - Guiding Principles reached the lowest mean score, with only 3.67 after the third group. The lesson plans’ mean score for item 5 was the lowest score among all items in each group. It can be found that the mean score for group 1 M1(SD) = 3.16(0.86) declined to M2(SD) = 2.78(1.18) for group 1 and dramatically increased M3(SD) = 3.67(0.84) for group 3 (as shown in Table A1 in the Appendices).

Moreover, we conducted the Non-Parametric Kruskal-Wallis Test for each factor to investigate what factors the difference might have appeared. Table 6 shows the analysis focus on every factor based on EDPLPR, indicating that all factors significantly differ based on the three groups. In addition, we provided the median of each factor, which differs from 4 and 5, as summarized in Table 6. Moreover, as summarized in Table 6, it can be identified that the effect size varies from medium to large effect, indicating that teachers who have been acquainted with design heuristics are likely to implement the principles provided in the heuristics into the lesson-planning practices.

Table 6. Independent samples Kruskal-Wallis test based on EDPLPR factors.

Factor	Median	χ^2	df	p	ε^2
Guiding Principles (LC)	4	46.5	2	<.001	0.2553
Defining the Problem (DP)	5	11.8	2	.003	0.0648
Idea Generations (IG)	4	48.5	2	<.001	0.2663
Designing and Construction (DC)	4	45.1	2	<.001	0.2477
Design Evaluation and Redesign (DR)	4	33.0	2	<.001	0.1816
Evaluation and Reflection (ER)	4	30.4	2	<.001	0.1670
Additional Components (AC)	4	30.8	2	<.001	0.1692

Significance at p < 0.05.

Furthermore, we continued the analysis by conducting post hoc analysis following the Dwass-Steel-Critchlow-Fligner pairwise comparisons for each factor based on three groups (see Table 7). Overall, the result presented in Table 7 suggested that all factors have roughly the same result for groups 1 and 2, indicating no significant difference for all factors. However, it should be noted that for groups 1 and 3, all factors have been proven to have significant differences. Likewise, Table 6 exhibits that all factors significantly differ between groups 2 and 3.

Table 7. Dwass-Steel-Critchlow-Fligner pairwise comparisons for EDPLPR.

Group		LC		DP		IG		DC		DR		ER		AC
		W	p	W	p	W	p	W	p	W	p	W	p	W	p
1	2	−0.698	0.875	0.593	0.908	−0.317	0.973	1.39	0.590	1.52	0.529	−0.381	0.961	−0.942	0.783
1	3	7.932	<.001	4.496	0.004	8.626	<.001	8.76	<.001	7.79	<.001	6.957	<.001	6.554	<.001
2	3	8.173	<.001	3.796	0.020	8.335	<.001	7.31	<.001	5.54	<.001	6.093	<.001	6.675	<.001

Significance at p < 0.05.

In summary, as presented in Table 6, there was a significant difference between each factor of EDP regarding groups based on the analysis of STEAM-EDP hybrid learning lesson plans (H02 was rejected). In addition, we considered measuring the magnitude of the difference between groups. Therefore, we measure the effect size of each factor by calculating the epsilon squared (ε²). As can be observed from Table 6, the value of epsilon squared (ε²) of each factor varied between 0.06 and 0.27. Based on Kirk (1996), the epsilon-valued, which scored 0.06 ≤ ε² <0.138, is considered a medium effect size, while ε² ≥0.138 is considered a large effect size. Therefore, it can be indicated that six EDPLPR factors (Factors LC, IG, DC, DR, ER, and AC) have a large effect size. In contrast, one of them (Factor DP) has a medium effect size, meaning the use of design heuristics has impacted teachers lesson-planning activities with a modest or not particularly strong contribution.

Although there is a significant difference for all groups, Table 8 shows that the significant difference only indicates Groups 1 and 3 (we reject H02b) and Groups 2 and 3 (we reject H02c). In addition, according to Table 8, we do not reject H02a (accepted), which indicates there is no significant difference between group 1 and group 2.

Table 8. Hypothesis testing summary.

Hypothesis	Result
H01a	Do not reject H0 (Accepted)
H01b	Reject H0
H01c	Reject H0
H02a	Do not reject H0 (Accepted)
H02b	Reject H0
H02c	Reject H0

We found that there was a statistically significant difference between teachers’ scores based on individual scores and EDP factors regarding groups (see Tables 4 and 6). It indicates a different contribution of design heuristics to the teachers’ lesson planning practices in each group. To sum up, according to the findings, we summarized the result of the testing hypothesis in Table 8. Based on the data analysis, both hypotheses, which related to the comparison of groups 1 and 2, were accepted. Meanwhile, it can be indicated that the hypothesis related to groups 1 and 3 and 2 and 3 were rejected as it is understandable that the mean and the median scores were slightly different for groups 1 and 2.

Discussion and implications

Based on the findings, there is a notable enhancement in teachers’ scores within three groups of implementations, as shown in Table A1 (Appendices). The implementation conducted in this study exhibits coworking activities with teachers through online workshops within the three groups that effectively contribute to developing design heuristics. This result aligned with a study conducted by Schallert et al. (2022), who reported the development of design heuristics, which implies the benefits of design research processes in supporting teachers’ practice of planning lessons. Through document analysis of lesson plans and scoring conducted in this study, we identified important practical experiences from teachers’ lesson planning activities. Moreover, this investigation is in line with a study conducted by Shareefa (2020), who revealed that document analysis of teachers’ lesson plans can be used to reveal the significance of practical implications. Therefore, the result corroborates that design research may positively contribute to enacting the quality of design heuristics and supporting teachers’ lesson planning activities.

The results indicate that the design heuristics could support teachers’ lesson-planning activities. For example, based on Table A1, the score of lesson plans based on the EDPLPR factors has risen from group 1 to group 3. The result exhibits an increase in teachers’ scores, achieving 106.20 after group 3 (as summarized in Table A1 in the Appendices). Additionally, Tables 1 and 2 show a significant difference in individual scores after three implementations, which implies a positive contribution of design heuristics. It can be indicated that teachers were likely to follow the guidance and incorporate the key principles of the design heuristics in their lesson plan development. These findings are corroborated by two studies on using design heuristics to support teachers in developing lesson plans (Janssen et al., 2009; Schallert et al., 2022). The finding is also supported by Herro et al. (2019), who suggested clear guidelines for planning STEAM lessons to empower STEAM instruction in classrooms. Therefore, it is expected that the result might provide valuable insights into how design heuristics can contribute to teachers’ lesson-planning activities.

Meanwhile, the finding also revealed a need for further investigation into supporting teachers in developing and defining problems for STEAM activities. As reported in Table 5, six factors have been proven to hold a large effect size, representing that the development of design heuristics after three implementations significantly impacts the six factors. The results shed light on the fact that teachers who participated in this study were likely to incorporate the key principles of design heuristics into their lesson-planning practices. However, as reported in Table 5, we found that one factor, Defining the Problem (Factor DP), has a medium effect size that is noticeable but not strong enough to have a significant practical impact. The findings highlight that all teachers face similar challenges with regard to the factors that might lead to creating barriers to engaging in more creative behaviors and developing problems in STEAM activities. One potential factor that might impact the result, as indicated by a study by Leroy and Romero (2021), is that teachers often emphasize quick solutions and use existing methods. Therefore, this finding suggested the significance of future investigations on the Factor DP and its impact on teachers’ lesson-planning activities, focusing on defining and developing problems for STEAM activities.

Furthermore, the result indicated that the mean score for item 5 or LC 5 (as shown in Table A1 in the Appendices), which evaluated the variety of assessment approaches provided in the developed lesson plan, was consistently the lowest score among all items in each group. The mean score in group 1 declined in group 2 but increased in group 3 (as observed in Table A1). The finding is in line with the result reported by Dubek et al. (2021), which pointed out that teachers faced challenges in STEAM classroom assessment. Moreover, the result might align with a study by Herro et al. (2017), who discussed the need for assessment tools to assess students’ collaboration in STEAM lessons. However, the need was not merely to focus on evaluating students’ collaboration but also the approaches that could assist with obtaining a deeper understanding of the learning’s impact. As pointed out by Graff (2022), goal orientation is important in relation to design project activities and learning goals, which are closely related to evaluation activities. Thus, the findings revealed that teachers still had difficulties conducting assessments for STEAM lessons, which need to be explored more in future research. To sum up, there is a need for improvement in the design heuristics, especially in the design principles related to the variation of assessment strategies and goal orientation for lesson planning to ensure alignment with the intended learning objectives.

With regard to the novelty of the study, the result showed that teachers who participated in this study were likely to incorporate the key principles of design heuristics in their lesson-planning practices. Despite the study result alignment with previous findings, this study provides novelty which lies in several aspects of the valuable opportunity to employ design heuristics in assisting technology-enhanced learning by exploring opportunities in hybrid learning which appears suitable for facilitating STEAM activities (Boon, 2021; Borba, 2021; Linder, 2017; Moskal, 2017). Furthermore, as we were aware of the challenges that teachers may face in implementing pedagogical guidelines, we realized that employing pedagogical guidelines might make a positive contribution to assisting teachers in developing STEAM lessons. This study can fill the gaps by continuing previous studies on the development of pedagogical guidelines in developing lessons (e.g., Davis & Krajcik, 2004; Janssen et al., 2009; Schallert et al., 2022) to empower the implementation of STEAM lessons that previous studies have not explored.

The findings of this study have the potential to address this gap by providing pedagogical guidelines for implementing STEAM education, which can support teachers’ professional development programs. As proposed by Ting and Tai (2023), teachers need to be equipped with suitable resources for professional development problems. However, we found that there is a necessity for further exploration, especially in the variation of assessment strategies and the exploration of problems or context for STEAM learning. The result emphasized the significance of future exploration that defines and formulates problems for STEAM activities and its implications on teachers’ lesson planning. Therefore, this study harmonizes technological advancement and pedagogical practices within technology-enhanced activities by emerging hybrid learning into lesson planning and evolving a design approach into STEAM education.

Conclusion and future plans

The findings revealed how pedagogical guidelines as a form of design heuristics could be structured to assist teachers in developing STEAM lesson plans through design research activities. Moreover, the findings report how the design heuristics support teachers’ lesson-planning activities. The results documented that participating teachers incorporate the principles provided in the design heuristics in their practices. Therefore, this study provides a new contribution to the design heuristics supporting teachers’ lesson planning with the heuristics from the third prototype.

While our study has aimed to provide valuable insights into assisting teachers with lesson planning, it is important to acknowledge that there were limitations that could affect the generalizability of the findings. These limitations are mainly related to the data collection and the potential bias in our sample. Nevertheless, the study offers limited research findings relating to the Indonesian context. For instance, in the online workshop, we provided an exemplary example of a lesson plan developed based on the design heuristics that had the potential to influence teachers’ lesson-planning activities. The following limitation was that the participants of this study voluntarily participated. Thus, the sample exhibits uneven distribution across groups based on their demographic profiles, thereby lacking balance. Finally, we will continue to follow the collaborative design in developing pedagogical guidelines for supporting the relevancy of STEAM education in the Indonesian context. We will continue to contribute with teachers and teacher experts for consultation and building consensus on the quality of pedagogical guidelines in our design research project.

Author contributions

The first author designed this research, conducted and analyzed data, drafted and finalized the manuscript; the second and third authors supervised and designed the research as well as finalized the manuscript; the fourth helped in critically reviewing and finalizing the manuscript; the fifth author critically reviewed and assisted the data analysis process. All authors read and approved the final manuscript.

Consent to participate

All collected data is used only for research purposes, and the participants were informed about this research and asked for their consent.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The collected data and its analysis are included in the text.

Additional information

Funding

This work is funded by the Austrian Federal Ministry of Education, Science and Research (BMBWF) with reference number MPC-2022-06063 under the Ernst Mach Grants – ASEA-UNINET. Additionally, the publication funding is supported by Johannes Kepler Open Access Publishing Fund and the federal state Upper Austria.

Notes on contributors

Pasttita Ayu Laksmiwati

Pasttita Ayu Laksmiwati is a PhD student at the Linz School of Education at Johannes Kepler University, Austria. Her research spans STEAM education, which focuses on investigating how STEAM education can be implemented in classrooms with the integration of design thinking.

Zsolt Lavicza

Zsolt Lavicza is currently a university professor at the Department of STEM Education, Linz School of Education, Johannes Kepler University, Austria. He is working on numerous research projects examining technology and its integration into schools.

Adi Nur Cahyono

Adi Nur Cahyono is a lecture at Department of Mathematics, Universitas Negeri Semarang in Indonesia. He is the founder of the Mobile Math Trails Research Group (mathe.id) and is working on the development of MathCityMap Indonesia.

Mara Alagic

Mara Alagic is a professor in the School of Education and serves as the Graduate Coordinator for the Master of Education in Learning and Instructional Design (LID) program at Wichita State University. Her current interests are focused on STEAM design heuristics and transdisciplinary teams’ collaboration in convergence science projects.

Filiz Mumcu

Filiz Mumcu has been currently working as an associate professor at Manisa Celal Bayar University, Turkey. Her research focuses on computational thinking, computing education, STEM education, and teacher education.

Appendix

Figure A1. Design heuristics for engineering design process in hybrid learning.

Table A1. Descriptive statistics based on individual scores and EDPLPR factors for mean scores.

Categories			Mean (SD)
			Group 1	Group 2	Group 3
Individual Score			84.98(14.09)	85.24(17.97)	106.20(16.19)
EDPLPR Factors	Guiding principles	LC 1	4.35 (0.75)	4.61 (0.57)	4.87 (0.52)
		LC 2	3.98 (0.93)	3.94 (0.75)	4.56 (0.81)
		LC 3	4.02 (0.65)	4.00 (1.08)	4.56 (0.75)
		LC 4	3.87 (0.79)	3.53 (1.17)	4.30 (0.95)
		LC 5	3.16 (0.86)	2.78 (1.18)	3.67 (0.84)
		LC 6	3.51 (0.74)	3.29 (0.84)	4.53 (0.71)
	Defining the Problem	DP 1	4.38 (0.78)	4.43 (0.71)	4.66 (0.68)
		DP 2	4.25 (0.80)	4.37 (0.73)	4.63 (0.74)
	Idea generation	IG 1	3.85 (0.85)	3.65 (1.05)	4.65 (0.68)
		IG 2	3.78 (0.85)	3.73 (1.08)	4.66 (0.71)
		IG 3	3.78 (0.83)	3.73 (1.08)	4.65 (0.68)
	Designing and constructing	DC 1	3.55 (0.90)	3.61 (1.08)	4.53 (0.81)
		DC 2	3.58 (0.90)	3.73 (1.04)	4.53 (0.81)
		DC 3	3.58 (0.94)	3.78 (1.09)	4.52 (0.81)
	Design Evaluation and Redesign	DR 1	3.35 (0.93)	3.65 (1.07)	4.30 (0.88)
		DR 2	3.33 (0.96)	3.57 (1.10)	4.27 (0.94)
		DR 3	3.33 (0.96)	3.57 (1.08)	4.25 (0.94)
		DR 4	3.25 (0.95)	3.43 (1.08)	4.24 (0.94)
	Evaluation and Reflection	ER 1	3.40 (0.93)	3.41 (1.06)	4.22 (0.93)
		ER 2	3.36 (0.93)	3.35 (1.03)	4.23 (0.89)
		ER 3	3.38 (0.89)	3.33 (1.03)	4.18 (0.87)
	Additional Components	AC 1	4.38 (0.65)	4.39 (0.49)	4.54 (0.75)
		AC 2	3.55 (0.72)	3.37 (0.81)	4.25 (0.87)
		AC 3	3.78 (0.85)	3.71 (1.06)	4.41 (0.79)
		Average	3.69	3.70	4.42

Table A2. Descriptive statistics based on individual scores and EDPLPR factors for median.

Categories			Median
			Group 1	Group 2	Group 3
Individual Scores			89	91	118
EDPLPR Factors	Guiding principles	LC 1	5	5	5
		LC 2	4	4	5
		LC 3	4	4	5
		LC 4	4	4	5
		LC 5	3	3	4
		LC 6	3	3	5
	Defining the Problem	DP 1	5	5	5
		DP 2	4	4	5
	Idea generation	IG 1	4	4	5
		IG 2	4	4	5
		IG 3	4	4	5
	Designing and constructing	DC 1	4	4	5
		DC 2	4	4	5
		DC 3	4	4	5
	Design Evaluation and Redesign	DR 1	3	4	5
		DR 2	3	4	5
		DR 3	3	4	5
		DR 4	3	3	5
	Evaluation and Reflection	ER 1	3	3	4
		ER 2	3	3	4
		ER 3	3	3	4
	Additional Components	AC 1	4	4	5
		AC 2	4	3	4
		AC 3	4	4	5

Table A3. Engineering design process rubric lesson plans.

No	Items
Guiding Principles (LC)
1.	LC 1	The content knowledge, concepts, and skills selected for the lesson are in alignment with national curriculum standards.
2.	LC 2	The learning objectives are measurable and achievable within a specific period.
3.	LC 3	The lesson combines two or more disciplines that help students see the relatedness of the concepts.
4.	LC 4	The lesson includes an open-guided question(s) to make it more relevant for students.
5.	LC 5	There are a variety of assessment approaches provided in the lesson plan that are appropriate to the learning goals.
6.	LC 6	During this lesson, the teacher facilitates clear instructions that emphasize the importance of each phase.
Learning Cycle – Engineering Design Process (EDP)
EDP – Defining and delimiting the problem
7.	DP 1	The problem selected gives age-appropriate challenges for students.
8.	DP 2	The problem selected elicits and accommodates different strategies from students to solve the problem and gives students opportunities to apply their knowledge and skill(s) throughout the design process.
EDP – Generating ideas (IG)
9.	IG 1	The lesson encourages students to recall prior knowledge.
10.	IG 2	The lesson ignites students’ curiosity and accomplishes something of interest to the students.
11.	IG 3	The lesson leads to idea generation.
EDP – Design and construct (DC)
12.	DC 1	The lesson encourages students to design and construct a model or prototype.
13.	DC 2	The lesson successfully encourages students to engage in purposeful teamwork.
14.	DC 3	The lesson plan adequately allows students to solve the problem with multiple and creative approaches.
EDP – Design evaluation and redesign (DR)
15.	DR 1	The lesson engages students to test and evaluate their design.
16.	DR 2	The lesson allows students to reflect and see their failures and mistakes.
17.	DR 3	The lesson successfully engages students to improve their design based on their evaluation.
18.	DR 4	The lesson encourages students to distinguish levels of strategies and their effects on the prototype.
EDP – Evaluation and reflection (ER)
19.	ER 1	The lesson allows students to reflect on their improvement.
20.	ER 2	The lesson involves students communicating their design process and results by referring to the guiding question(s)
21.	ER 3	The lesson encourages students to defend their strategies and solutions.
Additional Components (AC)
22.	AC 1	The lesson provides students with opportunities to predict and discuss the criteria for the success of a model or prototype and the constraints that will be used to solve the problem.
23.	AC 2	Appropriate safety issues are addressed.
24.	AC 3	The lesson plan provides relevant references and resources that include multiple sources for content verification.

Table A4. Scoring criteria of EDPLPR.

Score	Description
5	Excellent
	All elements on the item are present, appropriate, and accurate. The plan can be implemented by another teacher as written.
4	Good
	Most of the elements on the item are present, appropriate, and accurate. The plan can be implemented by another teacher with a few modifications
3	Average (Nor good or poor)
	Approximately half of the items’ elements are present, appropriate, and accurate. The plan can be implemented by another teacher with modifications
2	Poor
	Few of the elements on the item are present, appropriate, and accurate. The plan can be implemented by another teacher by re-write the lesson.
1	Unacceptable
	Key elements of the item are not present, and the descriptions are inappropriate, incomplete, and inaccurate. The plan lacks coherence and cannot be implemented by another teacher.