Enhancing critical thinking in operations management education: a framework with visual-based mapping for interdisciplinary and systems thinking

ABSTRACT

An instructional intervention was introduced to enhance critical thinking skills, in the aspects of interdisciplinary and systems thinking, in an undergraduate Operations Management course at Nanyang Technological University, Singapore. The intervention included a newly developed visual-based mapping tool to induce deliberate divergent-convergent thinking (imagination and judgement). It was supported by a design-based problem as a formative assessment, with learning outcomes underlined by a set of assessment rubric. To evaluate its effect on the students’ critical thinking abilities, a quasi-experimental study was conducted with data collected from survey instruments, semi-structured interviews, and academic scores. The survey results indicate improvements in students’ abilities in consciously monitoring their cognitive activities, reflecting on their reasoning, and making self-assessment (self-regulation) as well as in assessing credibility, relevance, and logical strength of information (evaluative reasoning). Results from the interviews and academic scores complemented the survey findings. The largely positive results suggest the effectiveness of the intervention.

KEYWORDS:

• Critical thinking
• interdisciplinary thinking
• systems thinking
• mapping tool
• operations management

1. Introduction

Critical thinking is concerned with the ability to engage in purposeful and self-regulatory judgement to comprehend a problem innovatively, and to develop effective solutions for the problem (Abrami et al., 2008; Sosu, 2013). Many studies have emphasized the significance of critical thinking skills in preparing students to cope with future challenges in both work and private lives (e.g. Dyck, Walker, Starke, & Uggerslev, 2012; Facione & Facione, 1996; Schneider-Cline, 2017).

The literature has yet to reach a strong consensus on the definition of critical thinking (Abrami et al., 2008; Calma & Davies, 2021). Studies have shown that its definition and what constitutes critical thinking differ between disciplines (Bezanilla, Fernández-Nogueira, Poblete, & Galindo-Domínguez, 2019; Jones, 2007; Moore, 2011). Thus, to be specific, this study aims to enhance critical thinking in Operations Management (OM) education in terms of improving interdisciplinary and systems thinking. (More discussion on the need for such thinking skills in OM education will be provided in the next sub-section.) Interdisciplinary thinking is of growing importance in today’s rapidly changing and interconnected world, where solutions to problems are often found beyond the confines of a specific discipline (Wang, 2019). Consideration of diverse viewpoints through an interdisciplinary perspective can improve critical thinking in terms of how one interprets information (Wang, 2019). Matthews et al. (2021) argued that interdisciplinary research and teaching will grow in importance as humans and computers become integrated in the Fourth Industrial Age. As for systems thinking, there are also many definitions (Arnold & Wade, 2015) but most agree that it involves the consideration of how a system’s components interrelate and interact with one another to achieve a specific purpose. Studies have shown that critical thinking can be improved through inclusion of interdisciplinary (Howlett, Ferreira, & Blomfield, 2016; Hubbard, 2021; Moirano, Sánchez, & Štěpánek, 2020; Newell, 1996) and systems thinking (Forrester, 1994; Grohs, Kirk, Soledad, & Knight, 2018; Jackson & Hurst, 2020) in teaching and learning.

Evidence has indicated that such skills can be acquired when suitable instructional guidance is provided. Improvements of interdisciplinary thinking skills in higher education resulted from interventions were reported, including Zimmerman, Lester Short, Hendrix, and Timson (2011) (health care), Cowden and Santiago (2016) (biochemistry), and Kuo, Tseng, and Yang (2019) (human-computer interaction). Other works include Mebert et al. (2020), which introduced an inter-institution, interdisciplinary student project that incorporated many existing best practices in student engagement. Hubbard (2021) studied disciplinary literacy teaching to support students in reading literature across STEM disciplines. In addition, Ison (1999) reported the benefits of incorporating systems thinking in higher education. Critical thinking has also been studied for Business Management education and recent works include the following. Gatti, Ulrich, and Seele (2019) implemented business simulation games in the teaching of sustainable development; Farashahi and Tajeddin (2018) carried out a comparison study on learning outcomes, cases studies and simulations in business education; D’Alessio, Avolio, and Charles (2020) studied the impact of critical thinking in an Executive MBA program; Janamanchi (2020) reported how student engagement and student learning were improved by incorporating critical thinking practices into an undergraduate Production and Operations Management class.

The above studies considered curriculum and instructional re-design to improve critical thinking ability. Besides curriculum changes, there exists a need for a problem-solving technique that can be applied by students. In addition, instructors need a tool that can be employed in class to demonstrate interdisciplinary and systemic interactions. This study aims to address these needs. Specifically, we introduced a newly developed instructional approach consisting of a visual-based mapping tool. During mapping, students apply a thinking model to guide their divergent-convergent thinking. To put it simply, divergent thinking equates to imagining various possibilities and interactions while convergent thinking involves judging their relevance and viability (Isaksen, Dorval, & Treffinger, 2010). By applying divergent and convergent thinking separately, one can better focus on generating ideas and then evaluating their suitability. The mapping tool and thinking model are supplemented by a design-based formative assessment, assessed by a set of course-specific rubric that aligns with the intended learning outcomes. Furthermore, the mapping technique can also be used for teaching to illustrate interdisciplinary and systemic concepts.

The research question for the study is as follows: Does the explicit and strategic incorporation of the aforementioned instructional approach lead to a significant improvement in the students’ critical thinking abilities in solving OM related problems?

Need for critical thinking in operations management education

OM refers to the administration of practices to utilize limited resources (e.g. materials, labor and productive capacities) efficiently with the objective of maximizing profit or minimizing cost. Applications of OM is wide, from managing daily life processes, such as queues at service counters (Queuing Theory) to large-scale global supply chains. OM sub-disciplines are diverse, relating to different industrial sectors and spanning across departments of a typical institute of higher education (including Business and Engineering). The sub-disciplines include Logistics and Supply Chain Management, Industrial & System Engineering, Process Management, Project Management and Hospitality Management.

The motivation behind this study is the need for critical thinking skills in OM education. First, OM problems usually comprise interrelated components or sub-problems that can be complex because of conflicting objectives among stakeholders. Thus it is important to incorporate a system-wide view of the various interactions and trade-offs. Janamanchi (2020) stated that systems thinking is a ‘fundamental and powerful concept’ for the Production and Operations Management undergraduate course. Second, as OM usually relates to many other disciplines (e.g. technology, psychology and marketing), one often needs to consider OM problems from a multidisciplinary viewpoint. The trend of businesses shifting from profit-focus to incorporate social, economic and environmental concerns further emphasizes the importance of such perspective (Jacobs & Chase, 2020). Third, OM is rapidly evolving, and practitioners need to adapt to new practices (e.g. sharing economy and technologies like Internet of Things); possessing an interdisciplinary mindset will facilitate one in finding innovative solutions. There has also been more focus on innovation in business management curriculum in recent years (e.g. Eggers, Lovelace, & Kraft, 2017; von der Heidt, 2018). Hence OM students need to develop interdisciplinary and systems thinking to enhance their ability to solve new and emerging problems.

These capabilities can largely be encapsulated in the skill of critical thinking. The definition of critical thinking offered by Scriven and Paul (2020) (largely based on the cognitive domain in Bloom, 1956) is most fitting to this study: Critical thinking is the intellectually disciplined process of actively and skilfully conceptualizing, applying, analysing, synthesizing, and/or evaluating information gathered from, or generated by, observation, experience, reflection, reasoning, or communication, as a guide to belief and action. Aligning this definition to interdisciplinary and systems thinking, students should acquire not only the ability to conceptualize bits-and-pieces of OM knowledge, but should also be able to apply, analyse, synthesize and evaluate constituent elements or parts of OM.

Using physical distribution (i.e. distribution of physical goods) as an example, the motivations for critical thinking is illustrated in Figure 1.

• Systemic view: Managing distribution systems requires evaluation of trade-offs between decision variables, e.g. freight transport (speed, cost, etc.), inventory level (quantity, location, etc.) and facilities (how many, size and location of plants, warehouses, etc.). Furthermore, conflicting aims between stakeholders (suppliers, manufacturers, retailers, consumers, etc.) exist within the system.

• Interdisciplinary: Decisions tend to be related to other disciplines, e.g. marketing strategy, product design, political, environmental, and cultural considerations, as well as emergence of new technologies.

• Fast-evolving: Physical distribution is currently subject to many of the rapid changes caused by digitalisation, sharing economy and e-commerce logistics.

Figure 1. Need for critical thinking for physical distribution.

As illustrated above, there is a clear need for more research on how best to address the above gaps in OM teaching and learning.

Purpose of study

The objective of this study is to develop an instructional approach that enhances critical thinking in the aspects of interdisciplinary and systems thinking. The approach comprises three components: (1) Visual-based mapping tool combined with divergent-convergent thinking model; (2) design-based formative assessment; and (3) assessment rubric that underlines the learning outcomes

We also conducted a study in an authentic learning setting with the aim to evaluate the effectiveness of the approach. We implemented the approach in an undergraduate course Distribution and Warehousing, which is Year-4 (final-year) course in the Maritime Studies program at Nanyang Technological University, Singapore.

2. Approach

Development and related literature

In this section, we describe the development of the approach and discuss the related literature that supports the design decisions at each development stage. Subsequently, we discuss how our work contributes to the literature.

Debates on how to teach or train critical thinking skill have spanned decades. The main argument focuses on whether it is more effective to provide specific courses on critical thinking, compared to integrating it into regular course instruction. With this objective, Abrami et al. (2015) did a meta-analysis to identify effective approaches and they suggested one method to categorise existing approaches based on differentiated levels of instructors’ involvement, including self-study, dialogue, and authentic instruction. Following, Puig, Blanco-Anaya, Bargiela, and Crujeiras-Pérez (2019) provided a systematic review and managed to categorize concurrent interventions into general approaches and immersion/infusion approaches. General approaches advocate that critical thinking shall be taught separately from the content of an existing subject while immersion/infusion approaches hold that critical thinking shall be integrated into standard subject matter. In contrast to infusion approaches, immersion approaches necessitate the explicit inclusion of critical thinking skills in the subject-matter curriculum. Since we are leveraging on an existing course, we adopted the immersion approach where we explicitly included the instruction on the application of divergent-convergent thinking during the process of visual-based mapping.

Our first task was to set the application requirements of the approach. First, it should be easily blended into the curriculum without the need to substantially amend its flow and content. Second, the emphasis and rigour on the domain knowledge must be preserved. That is, the new approach should not distract students from the course content. Rather, its application should be introduced in line with the curriculum and can be used to reinforce key concepts. Indeed, an emphasis on domain knowledge is found to be essential in facilitating critical thinking and creativity (Kirton, 2004; Simonton, 2000).

We embarked on the development by exploring high-level thinking models in the literature, with the aim of employing or adapting them as a guiding framework for students; these include Elder and Paul (2020), Isaksen, Dorval, and Treffinger (1999), and Sharunova, Wang, Kowalski, and Qureshi (2020). Among these models, we found the fundamental concept in Isaksen, Dorval, and Treffinger (1999) best meets our objective. This model originated from Osborn (1952) and had undergone several refinements since its inception. Their concept of deliberate alternation between divergent and convergent thinking (imagination and judgment) best aligns with our identified needs in OM education. That is, establishing interdisciplinary and systemic interrelations that require imagining related problem parameters and then judging their relevance to the problem. The concept is also supported by neurocognitive research; for example, Zhang, Sjoerds, and Hommel (2020) studied the neural correlates of divergent-convergent thinking, with findings derived from specific cortical brain-activation patterns. Other applications that rely on divergent-convergent thinking framework include Cheng (2019), Sadak, Incikabi, Ulusoy, and Pektas (2022), and van Hooijdonk, Mainhard, Kroesbergen, and van Tartwijk (2020). Furthermore, when compared to most other models, the underlying thinking framework is adequately concise and adaptable to a typical OM curriculum.

Besides the thinking model, it is essential for students to be equipped with a tool that they can apply to formulate, organize, and structure the related parameters during problem-solving; this is particularly important for our emphasis on systemic and interdisciplinary approach. We developed a visual-based mapping tool for this purpose. Well-known mapping tools include mind mapping (Buzan & Buzan, 1993 and concept mapping (Novak, 1996). The literature offers considerable support for such tools. For example, concept mapping results in meaningful learning by assimilating new concepts and propositions into existing cognitive structures (Li, Hwang, Chen, & Lin, 2021; Sieben, Heeneman, Verheggen, & Driessen, 2021). Concept mapping, mind mapping and other such tools are largely employed in teaching and learning for organizing, structuring and visualizing information, while less focus is placed on problem-solving. In contrast, we deployed a mapping technique to support problem-solving in view of system-wide and interdisciplinary interactions (although the developed technique can also be employed as a teaching tool to demonstrate interrelation of parameters). We named it Relationship Mapping (RM). Like other mapping tools, RM involves the construction of ‘linkages’ between ideas, concepts or parameters. However, RM is less hierarchical (unlike concept maps) and concepts or ideas do not ‘branched out’ of a central theme (unlike mind maps). Rather, RM focuses on establishing either cause-and-effect or inter-dependency connections (more on how RM is performed will be described in the next section).

RM complements the thinking model (which is a general approach) as it enables a more directed and discipline focused procedure during problem-solving. We devised an approach on how students apply divergent-convergent thinking during the mapping process. We adopt the Four-Step Learner’s Model (mainly by Puccio, Mance, & Murdock, 2010), which is a version of the model of Isaksen, Dorval, and Treffinger (1999), due to its use of easy-to-understand plain language. The model consists of four main steps: Clarify, Ideate, Develop and Implement. Students were taught to follow these steps during mapping to activate the divergent-convergent alternation. Figure 2 illustrates the iterative application of RM and Learner’s Four-Step Model. More details of this approach and its implementation can be found in the next section.

Figure 2. Iterative use of relationship mapping and learner’s 4-step model.

The above approach was supplemented by a practice exercise for students. In this respect, we incorporated a design-based formative assessment. Merits of design-based assignments have been extensively reported in the literature. Such assignments provide students with opportunities to think about their own critical thinking and creative processes (Popat & Starkey, 2010; Ulger, 2018). In our implementation, students were tasked to propose a distribution network design for a hypothetical company. We also developed a set of assessment rubric to measure the learning outcomes that align with the course and with emphasis on critical thinking. To meet this objective, we constructed the rubric by adapting the scales of California Critical Thinking Skills Test (CCTST) (Facione, 1990; Facione, Facione, Blohm, & Giancarlo, 2002); see Ahern, Dominguez, McNally, O’Sullivan, and Pedrosa (2019) for a review of assessments of critical thinking skills. The CCTST was chosen because it encompasses all the key thinking skills that we consider important in the OM context; further, it can be fittingly adapted to our approach.

As stated earlier, prior literature largely considered curriculum and instructional re-design to improve critical thinking ability and there is a lack of work in developing problem-solving tools. Our study contributes to the literature on critical thinking in that we developed a visual-based mapping approach that focuses on problem-solving, with the purpose of enhancing interdisciplinary and systemic thinking. (As stated earlier, existing mapping tools are mainly used for organizing, structuring and visualizing information, with much less emphasis on problem-solving.) Aligning with the definition of critical thinking by Scriven and Paul (2020) that was stated earlier, the collective intent is to allow students to exercise their cognitive skills related to apply (i.e. applying knowledge to real situations), analyze (breaking down ideas and determining how they are interrelated), synthesize (putting ideas into a new whole) and evaluate (making judgments internal evidence and external criteria). To our knowledge, the application with divergent-convergent thinking is the first attempt in the literature that explicitly incorporates a thinking framework in the application of a mapping tool. We also demonstrate how we implemented other instructional methods (design-based assignment and assessment rubric) to support the mapping approach in an authentic learning setting. This study also contributes to OM education by demonstrating that critical thinking skills can be improved by a simple instructional framework that can be easily blended into the curriculum.

Application

RM focuses on establishing linkages through interdependencies between problem parameters or factors, which can be either cause-and-effect or inter-dependency (represented by single- or double-headed arrows, respectively). Like concept mapping, one may include linking words or phrases to describe the linkages. During mapping, students also apply the Learner’s Four-Step Model to activate deliberate divergent-convergent alternation in their thinking.

As mentioned, RM is not performed hierarchically, unlike mapping techniques that start with higher-level concepts. This is because OM problem-solving is usually not intuitively performed in a ‘top-down’ hierarchical manner. For example, an OM problem may arise from an operation in a company’s department, but the solution may involve other departments (e.g. marketing, production and human resource), external factors (e.g. technology and government policies), or other external entities (e.g. other companies, other industrial sectors, or the authorities). When solving the problem, the natural cognitive process is to first consider the most related or ‘controllable’ parameters (control variables) before considering other ‘less controllable’ ones. Clearly, a strict top-down hierarchical approach is usually not applicable. However, it does not mean to say that RM application is without any structures or procedures. In classroom settings, we recognize that a well-design procedure is important as there is a need for students to be taught a structured way of mapping. On the other hand, the mapping process should not be too restrictive such that it would limit creativity.

We formulated the mapping procedure to align with the intuitive process of problem-solving. The parameters that are related are grouped into domains. In our implementation in Distribution and Warehousing, the respective domains were taught in class as part of the original curriculum and not specifically introduced for the mapping. There are a total of four domains, and collectively, they should encompass all key parameters involved in typical distribution problems. The first domain Logistics Drivers consists of the control variables most relevant to distribution. Indeed, these are the main planning parameters in distribution. Other domains involve the less related parameters, namely (in the order of decreasing relatedness): Cross-Functional Drivers, Intra-Business Functions and External Factors. The four domains are summarized below:

• Domain 1: Logistics Drivers (control variables of Freight Transport, Inventory and Storage Facility, and their interrelation with customer service).

• Domain 2: Cross-Functional Drivers (non-logistics but intra-company functions that are closely related to distribution, including pricing, sourcing and information).

• Domain 3: Intra-Business Functions (intra-company functions that are less related to logistics than those in Domain 2, which could include sales and marketing, production, research and development, and product design).

• Domain 4: External Factors (which could include political, economics, societal, environmental, technological, and legal factors).

The mapping begins with Logistic Drivers and then move on to the other domains in a sequence of decreasing relatedness. Parameters and inter-parameter linkages are identified and added to the map as one progresses through the domains; inter-parameter linkages can be intra- or inter-domain. Note that the parameters are pre-determined for the mapping in the first two domains Logistics Drivers and Cross-Functional Drivers. However, there is no fixed parameters in the other two domains (Intra-Business Functions and External Factors); thus, there tends to be a greater need to apply divergent-convergent thinking more actively for these domains to identify relevant parameters and their linkages. In particular, the interdependencies in External Factors are most open-ended and therefore, it requires the most amount of divergent-convergent thinking.

The above procedure of applying the domains in sequence coincides with the deployment of the mapping as a teaching tool, where the domains were sequentially introduced in accordance with the course topics to illustrate the interdisciplinary and systemic relationships. Specifically, we started the course with logistics planning (Logistics Drivers) and then introduced more planning considerations (related to Cross-Functional Drivers and Intra-Business Functions), and finally presented the ‘big picture’ of distribution (relating to External Factors). Figure 3 shows an application of RM in studying the problem of setting up ‘self-service pick-ups’ for e-commerce deliveries.

Figure 3. An example of RM application (self-service pick-ups for e-commerce delivery).

As stated earlier, the approach also included a design-based take-home assignment. It requires students to design a distribution network for a hypothetical chocolate manufacturing company that plans to enter the Singapore market. Key features of the company were described, including the company’s core competencies, its popular product range, typical customer profiles, sales seasonality, and marketing strategy. To make the problem rather open-ended, we did not describe these features in detail. Students were tasked to describe a ‘high-level’ recommendation of the distribution system. By ‘high-level’, they need not provide a detailed plan but an aggregated description of a network configuration that can address the company’s challenges and operating characteristics. Students were required to state any underlying assumptions for their recommendations. They were also encouraged to perform their own research to substantiate their arguments. Furthermore, innovative ideas (rather than limiting to those covered in class) were strongly encouraged. We conducted a post-assignment feedback session for students where good-quality answers were shared while areas that warrant improvement were highlighted. By doing so, a key message to the students was that there is no ‘standard’ solution to the problem.

As stated earlier, the assessment rubric are based on the California Critical Thinking Skills Test (CCTST). The scales in CCTST criteria comprises Analysis, Inference, Evaluation, Induction, Deduction and Overall Reasoning. As CCTST is a discipline-neutral assessment tool, we converted its measuring scales into learning outcomes that encompass the course-specific skills. Furthermore, in the usual spirit of outcome measurement, the student actions specified in the rubric must be observable and measurable. We also put much emphasis in ensuring the rubric are unambiguous and easily understood by students. The main adaptations are summarized below:

• Analysis and Inference in CCTST were combined into a single outcome Interpretation and Analysis of Problem; this reduces the number of outcomes, as both are similar to some extent in that they measure the initial understanding and investigation of the problem. Furthermore, this combined outcome is simpler for students’ comprehension.

• In short, Induction refers to making broad generalizations based on specific observations and in the presence of uncertainty; Deduction refers to one making conclusion based on an assumed truth of a set of beliefs in precisely defined contexts. To serve their intended purpose and to simplify them for students’ understanding, the emphasis on inferring and concluding were embedded in two criteria: Recommendation(s) and Limitation(s). In summary, Recommendation(s) refers to making recommendation(s) by inferring and/or concluding based on available information. Limitation(s) refers to identifying existing/potential limitation(s) and risk(s) by inferring and/or concluding based on available information and evidence.

• Overall reasoning: This outcome refers to a ‘sustained, focused and integrated application of other scales’. To make it clearer, we further added that it ‘refers to the integration of all ideas into consistent argument(s) and recommendation(s) without contradiction’.

Table 1 summarizes the list of outcomes and the associated desired actions and performance (each measured by a four-point scale).

Table 1. A summary of outcomes.

Interpretation and Analysis of Problem

Understanding and defining of problem. Identifying issues (with justification) based on appropriate evidence and/or valid arguments.

Evaluation

Evaluating credibility, suitability and logical strength of information that are used for analyzing or solving the problem.

Recommendation

Developing effective/efficient solution using unambiguous arguments and justification. Identifying/inferring/concluding interdependencies and trade-offs between planning parameters of recommendations.

Limitations

Identifying/inferring/concluding/justifying both existing and potential limitation(s)/risk(s), and possible future modification(s) that are needed over the intended lifespan of the recommendation.

Overall reasoning

Integrating all ideas and reasoning into convincing argument(s) and recommendation(s) without contradiction.

3. Evaluation of approach

Method and participants

Distribution and Warehousing, as the title suggests, consists of two parts – the first 8 weeks of classes focuses on the distribution (distribution of physical goods) while the remaining 5 weeks covers warehouse management. The approach was only implemented in distribution, as the warehousing part was taught by another instructor who did not participate in the study. The two parts are distinct, as distribution covers ‘big picture’ of physical distribution while warehousing encompasses the operations within the warehousing facility. We evaluated our approach by measuring and comparing students’ performances in an experimental group against a comparison group.

The comparison and experimental groups were enrolled in different semesters because it was not feasible to conduct strict experimental design in the same semester. The study followed the ethics approval procedure of the university and each student who agreed to take part have to sign an informed consent form. Although we do not foresee any significance of the sequence in the studies of the groups, the comparison group was conducted first to allow more time for the preparation of the intervention for the experimental group. We note that there is certain degree of inherent experimental control of the groups. The students from both groups were all from the Maritime Studies program. In each group, majority were in their final year of studies in the four-year program while the rest were third-year students. Being in the same program, the students were of similar academic background. They were admitted into the program based on the same admission criteria and hence they tend to fall within a specific range of academic proficiency. In addition, being in the same program, they underwent the same higher learning education as they completed mostly the same courses before enrolling in this course.

Majority of the students in each group (more than 95%) were aged between 22 and 25 years old. The students were generally receptive to critical thinking. This is reflected in the results of the critical thinking survey and from the interviews (which will be presented latter in the paper). Prior to this course, they had studied a variety of courses related to areas such as business and management (law, accountancy, ethics, etc.), economics, technology, and environmental sustainability. These studies would have provided them with some related multidisciplinary knowledge and the appreciation of interdisciplinary thinking.

Instructions for both the comparison and experimental groups were delivered through lectures and tutorials. In lectures, the subject materials were presented by the instructor and students could raise questions that could lead to short discussions. Tutorials were less formal where class discussions on the lecture topics were conducted. Each week of classes comprised a two-hour lecture and a one-hour tutorial session. Students who were enrolled in the first semester formed the comparison group. Although RM was not introduced for this group, the trade-offs and interdependencies between planning parameters were introduced and explained. Besides, there was no design-based take-home assignment. Subsequently, in the second semester, the instructor applied the approach with the enrolled students forming the experimental group. The students in the experimental group were introduced to the RM and Learner’s Four-Step Model; as stated earlier, the RM domains were introduced progressively in accordance with the curriculum. Students also worked on the design-based assignment with outcomes measured by the assessment rubric. There were 58 and 46 students in the comparison group and experimental group, respectively.

The mixed methods research was conducted, comprising three data sources (surveys, semi-structured interviews, and academic scores). The survey served as the main data source, supplemented by the academic scores while the interviews enabled qualitative uncovering of themes and explaining of findings obtained from the quantitative methods. We present them in the rest of this section.

Data sources

Survey

We developed a survey based on the findings from Critical and Reflective Thinking Scale by Sosu (2013), Measure of Critical Thinking and Problem Solving by Hwang and Chen (2017), Scale of Deep Approach to Learning by Laird, Shoup, Kuh, and Schwarz (2008), California Critical Thinking Skills Test by Facione, Facione, Blohm, and Giancarlo (2002), and Delphi Report of critical thinking by Facione (1990). The survey was formulated based on six constructs that are considered important for critical thinking in previous studies (e.g. Dyck, Walker, Starke, & Uggerslev, 2012; Facione, Facione, Blohm, & Giancarlo, 2002; Sosu, 2013); they are academic self-efficacy in critical thinking, analytical skills, evaluative reasoning skills, think-out-of-the-box, self-regulation, and problem-solving.

In the following, we explain each construct and the related works. Self-efficacy in critical thinking is concerned with one’s perceived competence in being able to think critically (Bandura, 1997). Analytical skills refer to the skills that comprehend the meaning and significance of varied statements and situations, and determine their intended and actual inferential relationships (Facione, 1990). Evaluative reasoning skills are concerned with skills that evaluate the credibility and relevance of varied statements and the logical strength of their actual and intended inferential relationships (Facione, 1990). Think-out-of-the-box refers to obtaining insight on an issue by transcending conventional ways of thinking and doing (Tsui, 2007). Self-regulation comprises three key components: cognition, metacognition and the willingness to engage the metacognitive and cognitive skills. Specifically, in this context, it refers to one’s willingness and capability in consciously monitoring one’s cognitive activities, reflecting on one’s reasoning, and making self-assessment of one’s conclusion and correction of mistakes (Dyck, Walker, Starke, & Uggerslev, 2012; Facione, 1990). Finally, problem-solving is related to one’s capability of effectively analyzing problems and coming up with solutions (Hwang & Chen, 2017). The research team, which encompassed expertise in both OM and learning sciences, worked together to ensure the survey’s relevance to the OM discipline, course curriculum and the instrument’s objective. To further affirm its face validity, we seek feedback from departmental colleagues in the fields of OM and learning sciences. Refinements were subsequently made to some items, mainly to improve the ease of interpretation by respondents. To reduce possibility of self-report bias, the survey was anonymous and there was no request for any personal information. The survey questionnaire contains 30 questions pertaining to the above constructs. Each question is based on a five-point Likert scale. We list the questionnaire in Table 2.

Table 2. List of questions in survey.

Analytical skills (1) I analyse the basic elements of the information given, such as examining a particular case or situation in depth and considering its components. (2) I include diverse perspectives when analysing the information given. (3) I explore the assumptions of the information given. (4) I seek important patterns and details about the information given. (5) I relate the information at hand to prior knowledge. (6) I establish links between different components/premises of the information given. (7) I identify causal relationships among the components in the information given.

Evaluative reasoning (1) I evaluate the value of the information given, including its premises, arguments, specific details, and methods and so on. (2) I evaluate the strengths and weaknesses of other people’s views on a topic or issue. (3) I evaluate the credibility of every detail in the information given. (4) I weigh competing points of view. (5) I seek to find a good argument that challenges some of my firmly held beliefs. (6) I justify the choices that I make in every important step.

Problem-solving (1) I apply the theories or concepts that I have learnt to practical problems or in new situations. (2) Before solving a problem, I seek to get a clear idea about the problem at hand. (3) Before solving a problem, I look for the cause of the problem. (4) Before solving a problem, I seek to identify the type of the problem first. (5) When solving a problem, I innovate the strategy and the process for solving the problem. (6) When solving a problem, I compare each possible consequence caused by different solutions.

Self-efficacy (1) This course makes me more confident to develop my own approach to solve distribution problems. (2) This course makes me more confident that I can identify what is important in solving distribution problems. (3) This course enables me to think more critically for decision-making in distribution problems.

Self-regulation (1) Through this course, I engage in a process of self-discovery about the meaning of distribution management. (2) Through this course, I reflect on what I think is important in distribution management. (3) Through this course, I think more critically about the procedures needed for effective distribution management. (4) Through this course, I think more critically about the elements in every decision-making in distribution.

Think-out-of-the-box (1) This course helps me think about nonconventional ways of solving distribution problems. (2) This course helps me think about the purpose of distribution management from different points of view. (3) This course helps me analyze distribution management problems from different perspectives. (4) This course helps me develop solutions for distribution problems from different perspectives.

In the following, we introduce the hypotheses based on interrelationships among the constructs. The strengths of these interrelationships between the comparison and experimental groups were compared with the objective of determining the influence of our instructional approach on students’ critical thinking.

As people’s actions, motivation, and affective states are based on their own beliefs (Bandura, 1997), their self-efficacy in critical thinking can largely affect the specific critical thinking skills (Dyck, Walker, Starke, & Uggerslev, 2012). Students with strong self-efficacy can freely practice their critical thinking skills without worrying about being criticized by others (Tsui, 2007). Refer to Pampaka et al. (2018) for more discussion of how academic self-efficacy is translated into specific academic performance. Therefore, we developed the following hypotheses:

H1a.

Self-efficacy is positively associated with analytical skills.

H1b.

Self-efficacy is positively associated with evaluative reasoning skills.

H1c.

Self-efficacy is positively associated with think-out-of-the-box.

When students are encouraged to engage in a self-discovery process of learning and to reflect on their reasoning process, their self-regulation are likely to be enhanced and their capabilities of problem-solving could be improved (Dyck, Walker, Starke, & Uggerslev, 2012; Tsui, 2007). Therefore, we proposed the following hypotheses:

H2a.

Analytical skills are positively associated with self-regulation.

H2b.

Analytical skills are positively associated with problem-solving.

H3a.

Evaluative reasoning skills are positively associated with self-regulation.

H3b.

Evaluative reasoning skills are positively associated with problem-solving.

H4a.

Think-out-of-the-box is positively associated with self-regulation.

H4b.

Think-out-of-the-box is positively associated with problem-solving.

Academic scores and semi-structured interviews

Scores for an in-class quiz question were compared between the comparison and experimental groups. It was carried out after all the topics (on distribution) were completed. For the experimental group, all the interventions were completed before the quiz. Both groups attempted the same question. Like the assignment, it required high-level recommendation of the distribution network design (of a hypothetical virtual restaurant that serves food only via deliveries). Given the much shorter time limit, the quiz question is of a smaller scale compared to the assignment. The description was shorter and fewer details were provided. Students were instructed to focus on the operational aspects in their answers, specifically on facilities, storage and transportation of the distribution system. However, it should be noted that the grading of both groups was all performed by the instructor. Hence inter-rater reliability was not assessed. We faced difficulty in enrolling a suitable second grader because this course is largely developed by the instructor and is thus much content specialized. Nevertheless, despite the limitation, we regard this data source as a reference in additional to the data obtained from the survey and interview.

Although the survey can offer important data related to how students’ critical thinking changes, it is still limited in providing an in-depth understanding into students’ experiences. Therefore we employed semi-structured interviews to complement the survey. In addition, the intent is to gather narrative information (qualitative data) on possible limitations of the current instructional design and to explain findings from the survey.

Eight students from the comparison group and seven students from the experimental group participated in each interview. To recruit interview participants, we applied the convenience sampling technique (also known as Haphazard Sampling or Accidental Sampling) based on students’ consent to take part in the study (Etikan, Musa, & Alkassim, 2016). The process followed the ethics approval procedure of the university. Each participating student signed an informed consent form and was reimbursed S$10 each for their participation. The interview, which lasted about 30 minutes for each student, was conducted by a researcher (second co-author) instead of the instructor himself; this is to ensure students feel more at ease in expressing their opinions. The guiding research questions for both the comparison and the experimental groups are the same; for instance, ‘to what extent do you think the course helped you improve your critical thinking related to this course?’ ‘Besides what the instructor taught in the class, can you think of things you did during the course that helped you improve your critical thinking?’

4. Results and discussion

Survey findings

We first tested the reliability as well as the convergent and divergent validity of the constructs. Table 3 shows that the composite reliability values of the constructs in both groups are all above 0.70. In addition, the Cronbach’s alpha values of the constructs in both groups are all greater than 0.70, except for self-efficacy construct in the experimental group (α = 0.69), which is close to 0.70. Thus, overall, we can conclude that the survey instrument used in the current study has adequate reliability. Furthermore, the values of the average variance extracted (AVE) for all the constructs in both groups exceed 0.50, which suggest sufficient convergent validity of the research model for both groups (Fornell & Larcker, 1981). In addition, the square root of the AVE for each construct in both the comparison and experimental groups is larger than the correlation between that and all the other constructs (see Tables 4 and 5); it indicates that the research models in both groups demonstrate sufficient discriminant validity.

Table 3. Reliability and average variance extracted (AVE) in measurement model of both groups.

	Composite reliability		Cronbach’s alpha		AVE
	C	E	C	E	C	E
Self-efficacy	0.87	0.83	0.78	0.69	0.69	0.61
Analytical skills	0.85	0.82	0.76	0.70	0.59	0.53
Evaluative reasoning skills	0.87	0.88	0.80	0.81	0.63	0.64
Thinking outside box	0.87	0.90	0.81	0.86	0.63	0.70
Self-regulation	0.87	0.87	0.79	0.79	0.62	0.62
Problem-solving	0.87	0.89	0.77	0.81	0.63	0.73

Note. C= Comparison group; E= Experiment group.

Table 4. Discriminant validity of comparison group.

	SE	ANA	EVA	TOB	SR	PS
Self-efficacy	0.83
Analytical skills	0.56	0.77
Evaluative reasoning skills	0.54	0.63	0.79
Thinking outside box	0.68	0.32	0.45	0.79
Self-regulation	0.73	0.46	0.59	0.65	0.79
Problem-solving	0.50	0.59	0.51	0.37	0.39	0.82

Note. Bold values in diagonal row are square roots of AVE of research constructs.

Table 5. Discriminant validity of experimental group.

	SE	ANA	EVA	TOB	SR	PS
Self-efficacy	0.78
Analytical skills	0.47	0.73
Evaluative reasoning skills	0.55	0.64	0.80
Thinking outside box	0.79	0.44	0.52	0.84
Self-regulation	0.71	0.61	0.52	0.65	0.79
Problem-solving	0.49	0.65	0.73	0.40	0.47	0.85

Note. Bold values in diagonal row the square roots of AVE of research constructs.

Next we performed the descriptive analysis. As shown in Table 6, the average values of the constructs in the experimental group are all greater than those in the comparison group. In addition, self-regulation shows statistical significance with p = 0.02 (<0.05) and with effect size = 0.53. As the standard deviations appears considerably different between the groups, we performed the two-tailed F test, which indicates that there is not enough evidence to reject the null hypothesis that the variances of the two groups are equal at the 0.05 significance level.

Table 6. Descriptive statistics of comparison and experimental groups.

	N		M (SD)
	C	E	C	E	Mean difference	Independent sample t-test
Self-efficacy	58	46	4.03 (0.53)	4.16 (0.40)	−0.13	t(102)= −1.32, p= 0.19
Analytical skills	58	46	3.84 (0.55)	3.91 (0.48)	−0.07	t(102)= −0.68, p= 0.50
Evaluative reasoning skills	58	46	3.73 (0.65)	3.88 (0.59)	−0.15	t(102)= −1.25, p= 0.22
Thinking outside box	58	46	4.00 (0.56)	4.16 (0.58)	−0.16	t(102)= −1.41, p= 0.16
Self-regulation	58	46	3.89 (0.58)	4.15 (0.49)	−0.26*	t(102)= −2.44, p= 0.02
Problem-solving	58	46	4.01 (0.61)	4.08 (0.55)	−0.07	t(102)= −0.62, p= 0.54

Note. C= Comparison group; E= Experiment group; *p<0.05.

The improvement in self-regulation could be attributed to the emphasis of divergent-convergent thinking during mapping, which prompted students to deliberately assess their own cognitive processes. In addition, applying the mapping could have caused them to consciously reflect on their reasoning and to make self-assessment, particularly when identifying the relevant parameters and establishing the linkages.

The other constructs were improved in the following ways. The mapping tool and design-based assignment provided opportunities for students to apply critical thinking, providing them with practices to reflect and articulate their arguments; it could have helped students to build up confidence in critical thinking when solving problems, which improves self-efficacy. Similarly, evaluative reasoning skills could have been enhanced by the mapping process when forming the linkages between parameters (i.e. evaluating logical strength of actual and intended inferential relationships); in addition, the outcome measure of Evaluation in the assessment rubric (evaluating credibility and relevance of information) emphasizes to students the importance of this intended skill. In addition, the divergent thinking that is applied during mapping could have induced Think-out-of-the-box behavior. The two other constructs (analytical skills and problem-solving) are more obvious skills that our approach aims to improve.

Based on the hypotheses formed, we compared the structural models of both groups in terms of the path coefficients between different constructs using the bootstrap t-test technique. Table 7 shows a significant difference between the groups in terms of the path coefficient of evaluative reasoning skills on problem-solving (H3b), indicating evaluative reasoning skills has a statistically stronger effect on problem-solving in the experimental group than in the comparison group.

Table 7. Comparison between comparison and experiment groups.

Hypotheses		Comparison group	Experiment Group	diff.abs	t	df	p	Sig.05
H1a	SE -> ANA	0.56	0.47	0.09	0.41	102	0.34	No
H1b	SE -> EVA	0.54	0.55	0.01	0.001	102	0.50	No
H1c	SE -> TOB	0.68	0.79	0.11	1.02	102	0.15	No
H2a	ANA -> SR	0.13	0.39	0.26	1.54	102	0.06	No
H2b	ANA -> PS	0.44	0.31	0.13	0.50	102	0.31	No
H3a	EVA -> SR	0.29	0.02	0.27	1.29	102	0.10	No
H3b	EVA -> PS	0.17	0.54	0.37	1.73	102	0.04	Yes
H4a	TOB -> SR	0.48	0.47	0.01	0.003	102	0.50	No
H4b	TOB -> PS	0.16	−0.03	0.19	0.91	102	0.18	No

Note. diff.abs= absolute difference; the row in bold indicates paths where comparison group significantly differs from experimental group.

The statistical significance suggests an improvement of students’ evaluative reasoning skills in the experimental group which in turn positively affected their problem-solving skills. Indeed, evaluative reasoning is an important ability in problem-solving (Dyck, Walker, Starke, & Uggerslev, 2012; Tsui, 2007). The improvement could be attributed to the instructional intervention because external intervention is typically needed to improve such skills (Tsui, 2007). The improvement could be due to more conscious self-assessing of information when applying RM during problem-solving. Additionally, it could be owed to Evaluation being included as an intended learning outcome in the assessment rubric, which emphasizes its importance to students when they solve the design-based assignment.

Academic scores and interview findings

From the average scores of the in-class quiz based on the four-point rubric, the mean (standard deviation) of the comparison group is 2.77 (0.160) and of the experimental group is 3.04 (0.189); the difference is statistically significant (p < 0.05).

The interviews were transcribed and analysed thematically, following an iterative process as described by (Braun & Clarke, 2012), where we relate the interview transcripts back and forth to the research hypotheses and survey constructs. The themes that eventually emerged from the transcripts centered around the students’ inclination towards critical thinking and the intervention’s influence on learning and its limitations.

Table 8 shows the summarized results for the comparison group. The students from the comparison group mostly agreed that the fundamental theories and content knowledge taught in this course contributed to their proficiency in the subject. However, some students responded that the lecturer-centered pedagogical style, with a lack of guidance and activities on critical thinking, was not conducive to their development of such capabilities and their motivation to think critically. A few students also stated that they often could not easily identify the intended and actual inferential relationships among different information. Interviews with the experimental group, as summarized in Table 9, largely supported the effectiveness of the instructional intervention. Most of the students recognized the greater emphasis on critical thinking and appreciate the better ‘big picture’ view of the subject that align with the curriculum flow. The interviews revealed that the students view RM as an important part of the course in understanding the ‘big picture’ of the course topics (which relates to interdisciplinary and systems thinking in this study’s context). They appreciate the organized way of problem-solving using the mapping.

Table 8. Summarized interview results for comparison group.

Examples	Code	Theme
“The course’s pedagogical style is lecturer-centered with a lack of content and activities on critical thinking …… it does not help or motivates the development of critical thinking capabilities.”	Lack of activities related to critical thinking	Lack of guidance and activities that motivates and induces critical thinking
“The instructor did not really provide guidelines and approaches in helping us think critically …… we relied on concepts learned from textbooks and our own understanding to solve problems.” “The relationships between ideas and concepts are sometimes difficult to grasp …… they are not straightforward when analyzing questions in quizzes and tutorials”	Lack of guidelines on how to think critically
“ … … most of the time, you don’t have the energy to go and think critically. It is quite difficult for us to initiate discussions when you are comfortable with the lectures and the supplying (of content knowledge)”	Lack of motivation towards critical thinking

Table 9. Summarized interview results for experimental group.

Examples	Code	Theme
Compared to other modules (taught by other professors), this module requires more critical thinking	Content emphasizes on critical thinking	Critical thinking in terms of “Big Picture” view enhanced by Relationship Mapping
“He asks us to identify what are the major problems the companies may face … .When he emphasizes the trade-offs (in managing distribution and warehousing)” “He often uses the Relationship Mapping, which is very good. Every time when you look at this (the map), you can understand the whole picture.”	“Big picture” view facilitated by Relationship Mapping
“For the quiz and the assignment, I think it’s all about critical thinking … The case studies in both quizzes and assignment are much applied.”	Critical thinking in solving practical problems in quiz and assignment	Effectiveness of the quiz and assignment as formative assessments
“To answer the (assignment and quiz) questions properly … in the very organized method, you have to think through what is the first thing you have to look at. (For instance) for the questions in some other modules, there are three different models (to answer them), you can answer them using either fully A, fully B, or fully C. But for his assignments, (the solution) can be a mix of A, B, and C. You can even create your own solution D.” “You have to justify (your answers) …… as there is no fixed solution. So you really have to think why it (the solution) is the best.”	Open-endedness of quiz and assignment
“ …… the class size of the class was little too large, and as a result, it was difficult to have sufficient one-to-one consultation in honing critical thinking skills.”	Prefer more individualized consultation on improving critical thinking skills	Limitations of the intervention
“ …… we prefer more interactions among classmates as well as with the instructor.”	Prefer more class interactions
“The course should include field trips to see real-life industry practice.”	Prefer more experiential learning

The interview findings are also consistent with the survey findings. In particular, the students also felt that the open-ended problems (with no fixed solution) offered by the quiz and the assignment is a key feature of the instruction. We infer that this led to more critical thinking in terms of divergent-convergent thinking. This could result in greater degrees of self-regulation and evaluative reasoning, which are the survey constructs that were found to be statistically significant.

However, the instructional intervention used for the experimental group is not without shortcomings. For instance, a few interviewed students pointed out that the size of the class was little too large for sufficient one-to-one consultation in honing their critical thinking skills. Students also preferred to have more interactions among classmates and with the instructor. Some students felt that the course should encompass elements of experiential learning such as field trips to connect the content knowledge with actual practice. One suggested that more modules related to critical thinking should be provided in their earlier years of studies.

Overall, the findings from the academic scores and the interviews reinforced the outcome from the survey findings that the instructional intervention was effective in improving the development of student critical thinking.

5. Conclusion

We developed an approach with the objective of improving critical thinking skills in OM education. The Relationship Mapping (RM), which is applied together with divergent-convergent thinking, helps students establish interdependencies between parameters when solving OM problems. A design-based assignment forms the formative assessment where students get to apply their problem-solving skills, with an emphasis on critical thinking that is outlined in the assessment rubric developed in the study. Our findings supported the research question in the affirmative, that the explicit and strategic incorporation of the aforementioned instructional approach led to a significant improvement in the students’ critical thinking abilities in solving OM related problems. The findings are summarized in the following:

• The average values of the constructs in the experimental group are all greater than those in the comparison group. In addition, the construct of self-regulation shows statistical significance.

• Evaluative reasoning skills has a statistically stronger effect on problem-solving in the experimental group than in the comparison group, thereby indicating that an improvement of students’ evaluative reasoning skills consequently enhances their problem-solving skills. It could be a result of the instructional intervention because external intervention is typically needed to improve such skills (Tsui, 2007).

• The experimental group’s average score of the in-class quiz is higher, with self-regulation showing statistical significance with p = 0.02 (<0.05).

• Findings based on quiz scores and from the interviews largely supported the effectiveness of the instructional intervention.

Although this study was performed in a specific discipline and course, the findings have some wider implications in higher education. The findings provide further evidence that instructional intervention using hands-on activities/tools and deliberate thinking approach could lead to more purposeful assessment of students’ own cognitive process. This aligns with similar findings in Yelich Biniecki and Conceição (2016), Hill (2005) and Sieben, Heeneman, Verheggen, and Driessen (2021). Specifically, the intervention’s emphasis on the ‘big picture’ context of a discipline led to a more conscious self-reflection and assessment (self-regulation) in applying imagination (think-out-of-the-box) and judgment (related to analytical skills and evaluative reasoning skills). This could subsequently enhance problem-solving skills, and in the process, build up related confidence (self-efficacy). From a broader viewpoint, this study provides encouraging results for educators who wish to enhance students’ critical thinking skills, by demonstrating that such skills can be improved by incorporating relatively simple instructions that can be easily embedded in the curriculum without considerably disrupting its content and flow. This study also shows how the integrated approach consisting of problem-solving technique, thinking model and formative assessment can be implemented.

Nevertheless, this study has the following limitations that are worth addressing in the future. First, the approach has only been implemented in one course, hence limiting the generalizability of the research findings; applications in other OM sub-disciplines (or in other disciplines) are needed to assess its suitability in other applications. Second, to achieve a more comprehensive evaluation of the approach, future studies should include a larger sample size and greater diversity of students in terms educational background, academic proficiency and educational level; this will also enable a more targeted assessment of how the approach affects different types of students. Third, for practical reasons, inter-rater reliability was not evaluated when we utilized the in-class quiz scores, thereby may undermine the credibility of this source of data; addressing this limitation will significantly improve the validity of this data source. Fourth, the limitations stated by the interview respondents (e.g. lack of experiential study trips and in-class interactions) highlighted the broader demands from students in improving their critical thinking skills; these issues would need to be addressed in future implementations. Fifth, self-report survey could be subject to personal biases, such as the tendency to provide socially desirable responses. In this study, we put in place mitigating measures to minimise its impact, which include respondent anonymity and adopting the same survey for both comparison and experimental groups; furthermore, the interview findings corroborated with the survey findings. In future studies, it would be worthwhile to employ more objective assessment tools to complement the existing instruments to further reduce the possibility of biases.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Correction Statement

This article has been corrected with minor changes. These changes do not impact the academic content of the article.

Additional information

Funding

This project is supported by the EdeX Grant from the Teaching, Learning and Pedagogy Division, Nanyang Technological University. This study had obtained approval from ethics clearance (IRB) ref. no. IRB-2018-04-002 from host institution Nanyang Technological University.