1,895
Views
1
CrossRef citations to date
0
Altmetric
Original Article

The demise of active learning even before its implementation? Instructors’ understandings and application of this approach within Japanese higher education

ORCID Icon & ORCID Icon

ABSTRACT

Active learning is the outcome of a series of higher education policy reforms conducted in the late 2000s and early 2010s in Japan. This approach gained momentum after it was prioritised in a comprehensive report produced by the Central Council of Education (an advisory board of the Ministry of Education, Culture, Sports, Science, and Technology). It was also emphasised in a project undertaken by this Ministry. Despite its popularity and the emergence of publications focusing on its definition and associated methods, some scholars have expressed concern that active learning is not well understood or appropriately applied by Japanese university instructors. However, to date, no large-scale empirical studies have been conducted to elicit their understanding and practice of active learning. The current study addressed this research gap through a nation-wide survey of university instructors. Its findings indicate that the scholars’ concerns are largely justified. Whereas the majority of respondents reported the use of instructional methods for implementing active learning recommended by the Central Council (e.g. project- or problem-based learning and cooperative learning), they tended to equate active learning with the mere use of these methods. In conclusion, we offer suggestions on how active learning should be understood and practiced by instructors.

Introduction

In light of the validation of active learning (AL) within empirical studies (Freeman et al., Citation2014; Kinoshita, Knight, & Gibbes, Citation2017), this approach has gained increased popularity within higher education worldwide (Taylor & Ku, Citation2011). Despite growing recognition of its efficacy, many instructorsFootnote1 remain unfamiliar with AL approaches (Michael, Citation2007; O’Grady, Simmie, & Kennedy, Citation2014) or are “uncertain about how to develop active learning exercises for their classes” (Graeff, Citation2010, p. 265). In this context, the current study examines how instructors understand and practice AL within Japanese higher education. An understanding of prevailing AL perceptions and practices is important because they influence instructors’ teaching approaches, which, in turn, influence students’ approaches to learning (Kember, Citation2009).

AL emerged as an approach for enhancing employability skills within Japanese higher education following a series of educational policy reforms conducted in the late 2000s and early 2010s. These policy reforms led to the introduction of the concepts of gakushiryoku (abilities that university graduates should possess) in 2008 and of sygyoryoku (literally “employability skills”) in 2009 (Kakuhou, Matsumura, & Hirata, Citation2011). This policy shift was arguably influenced by the Bologna Process that was initiated in Europe in 1999, with the aim of developing and enhancing employability skills (Ogata, de Weert, & Yoshimoto, Citation2007). The popularity of AL in Japan gained momentum after its significance was highlighted in a comprehensive report titled “Towards Qualitative Transformation of Undergraduate Education for the Future, released in 2012 by the Central Council for Education (an advisory board of the Ministry of Education, Culture, Science, Sports and Technology). Its significance was further highlighted in a national project funded by the Ministry of Education, Culture, Science, Sports and Technology on “Improving Higher Education for Industrial Needs” that was implemented from 2012 to 2015 and involved 146 universities (Ministry of Education, Culture, Science, Sports & Technology, Citation2012).

Some scholars, however, have pointed out that many Japanese university instructors appear to have an inadequate understanding of AL or that they apply it incorrectly (e.g. Mizokami, Citation2014; Nakasono & Tanigawa, Citation2018), in part because of the relative novelty of this approach. Kawano (Citation2016) suggested that because current university instructors received a lecture-based education when they were students, they do not have a concrete understanding of what AL is and how it should be practiced. Although there have been numerous publications that define it and explain its methods, particularly since the mid-2010s, these works are mostly based on the authors’ subjective opinions, anecdotal experiences, or findings from relatively small-scale studies, entailing the use of data derived from their own students and universities and at best from inter-university projects. Therefore, they do not clarify how AL is to be understood or applied by Japanese university instructors. The current study addresses this research gap, with a large-scale empirical study, to understand instructors’ understanding and practices, including assessment.

This article is organised as follows. We begin by reviewing the existing literature on AL definitions and practices. Next, we elucidate how AL had been defined and practiced in the context of Japanese higher education. Subsequently, we present our methodology and results, followed by a discussion that includes suggestions on how AL should be understood and applied by Japanese higher education instructors.

Literature review

Definitions

Despite the proposal of a variety of definitions for AL (O’Grady et al., Citation2014), engagement is evidently a key defining feature of this concept. According to Chi and Wylie (Citation2014), student engagement can be considered from multiple perspectives (i.e. motivational, behavioural, emotional, and cognitive) at different modes of AL (i.e. passive, active, constructive, and interactive).

Motivation refers to precursor attitude/interest in learning contents. As Barkley (Citation2009) explains, engagement is rooted in motivation. Motivation develops into cognitive engagement, which positively influences learning through behavioural and emotional engagement (Lee, Citation2014).

Behavioural engagement refers to the notion of participation, effort, and investment in tasks (Gibbs & Poskitt, Citation2010). Student participation can be defined as asking questions, making comments, and discussing with instructors and/or peers in class activities (Dancer & Kamvounias, Citation2005).

Emotional engagement is considered to influence academic performance through behavioural engagement. Emotional engagement refers to the relationships between teachers and peers and a sense of belonging (Wimpenny & Savin-Baden, Citation2013). Emotionally stimulating learning experiences are essential to the notion of co-production that supports genuine engagement (Carey, Citation2013).

Cognitive engagement is defined as the amount of effort and type of processing strategies used for learning. Achievement goals, self-regulation, perceived instrumentality, and, again, a sense of belonging are factors associated with cognitive engagement. Cognitive engagement is influenced by the achievement goals, comprising mastery and performance goals. Mastery goals aim to increase competence related to task mastery while performance goals involve competence relative to others (Smiley & Anderson, Citation2011). Self-regulation is the cyclical process of working towards a goal rather than a once-off effort for achieving goals (Walker, Greene, & Mansell, Citation2006). The concept of perceived instrumentality refers to student recognition of the importance of learning towards achieving future personal goals (Sedaghat, Abedin, Hejazi, & Hassanabadi, Citation2011). Sense of belonging is an aspect of both cognitive and emotional engagement, and a mediator of motivation and achievement (Akiva, Cortina, Eccles, & Smith, Citation2013). These kinds of engagement are interrelated: motivation is the beginning and base of engagement, and both behavioural and emotional engagements are prerequisites for cognitive engagement.

According to Chi and Wylie (Citation2014), there are also different modes of engagement. Graded from high to low levels, these include the interactive, constructive, active, and passive modes. The passive mode of engagement refers to listening to a lecture with attention and perhaps interest but without doing anything else. The active mode of engagement refers to learners’ engagement with instructional materials that “can be operationalized as active if some form of overt motoric action or physical manipulation is undertaken” (Chi & Wylie, Citation2014, p. 221). The constructive mode of engagement encourages learners to actively construct knowledge and demonstrate externalised outputs or products. The interactive mode of engagement refers to interpersonal activities (e.g. discussions, dialogues) that meet two criteria: “(a) both partners’ utterances must be primarily constructive, and (b) a sufficient degree of turn taking must occur” (Chi & Wylie, Citation2014, p. 223).

Among these engagement modes, students’ cognitive engagement in the learning process is underscored within AL (Taylor & Ku, Citation2011; Wilson, Pollock, & Hamann, Citation2007), typically through interactive cognitive activities (Misseyanni, Lytras, Papadopoulou, & Marouli, Citation2018; Taylor & Ku, Citation2011), entailing the “building, testing, and refining their mental models” (Michael, Citation2007, p. 43). This notion of rebuilding mental models stems from Piaget’s constructivist learning approach (Biase, Citation2019; Drew & Mackie, Citation2011), with which students actively reconstruct prior knowledge by continuously building new knowledge (Diamond, Koernig, & Iqbal, Citation2008). This cognitive process of AL is associated with deep learning that arguably takes place in the cognitive engagement stage (Ito, Citation2017), as opposed to surface learning (Revell & Wainwight, Citation2009; Tangney, Citation2014). Whereas surface learning is acquired through extrinsically motivated processes, entailing the accumulation of information, deep learning occurs through an intrinsically motivated process of meaning construction (Golightly & Raath, Citation2015). Diamond et al. (Citation2008, p. 118) observed that “[a] key outcome of deep learning is the ability to apply concepts acquired in one context to a variety of new situations.” Thus, deep learning entails the acquisition of various transferable, generic skills (Levant, Coulmont, & Sandu, Citation2016) that can be used in different contexts, such as critical thinking (Peters, Citation2011; Riley & Ward, Citation2017), problem-solving (Dadach, Citation2013; Rico & Ertmer, Citation2015), and communication skills (Kember & Leung, Citation2005; Laverie, Citation2006). These generic skills overlap with and can be developed into employability skills. Indeed, as affirmed by Drew and Mackie (Citation2011, p. 44), “one of the most significant underlying reasons offered for the current interest in active learning is as a response to changing economic demands” and the need to develop generic employability skills (Chan et al., Citation2018; Virtanen & Tynjälä, Citation2019).

Practices

From an instructor’s standpoint, AL is a pedagogy that encourages students to be actively engaged in learning, regardless of the types of instructional methods used. That is, AL is not simply a collection of specific methods; rather, it is a theory of learning (Drew & Mackie, Citation2011; Golightly & Raath, Citation2015) or a pedagogical methodology (Ní Raghallaigh & Cunniffe, Citation2013). However, several methods are widely used to promote AL, notably project-based (O’Grady et al., Citation2014) or problem-based learning (Hussain, Mamat, Salleh, Saat, & Harland, Citation2007; Rico & Ertmer, Citation2015) and collaborative or cooperative learning (Dadach, Citation2013; Riley & Ward, Citation2017). Both problem-based learning and cooperative learning entail a constructive approach in which students engage in “self-directed learning and then apply their new knowledge to the problem and reflect on what they learned and the effectiveness of the strategies employed” (Hmelo-Silver, Citation2004, p. 235). Problem-based learning and cooperative learning are considered helpful for developing generic skills and enhancing deep learning (Golightly & Raath, Citation2015; Laverie, Citation2006). They commonly include discussions and debates (Hurd, Citation2000; Wilson et al., Citation2007), presentations (Deeley, Citation2014; Revell & Wainwight, Citation2009), simulation (Golightly & Raath, Citation2015; Levant et al., Citation2016), role-playing (Dadach, Citation2013; Kember, Citation2009; Misseyanni et al., 2016; Riley & Ward, Citation2017), case studies (Hurd, Citation2000; Misseyanni et al., 2016), and the use of the case method (Rico & Ertmer, Citation2015). Instructional methods or approaches such as problem-based and cooperative learning are methods that can be elements of AL (Graeff, Citation2010). However, they may not constitute a pedagogical design or methodology in isolation because it is the entire process of learning within an educational programme and course (Harrison, Citation2010).

Assessment

Behavioural measures are most commonly used to assess AL outcomes. Accordingly, students take actions that demonstrate what they have learned (Graeff, Citation2010). Forms of assessment typically involve tests, theses, portfolios, reports, projects, exams, essays, and presentations (Dadach, Citation2013). Tools such as rubrics (Rico & Ertmer, Citation2015) and self-evaluation sheets (Deeley, Citation2014) are also used to facilitate assessment. AL is especially compatible with formative rather than summative assessments, as it promotes a constructivist approach that enhances deep learning (Rushton, Citation2005). Formative assessments often include feedback elicited through comments that can serve to encourage student engagement (Esterhazy & Damşa, Citation2017). Grosas, Raju, Schuett, Chuck, and Miller (Citation2016, p. 1596) argued that “feedback should become a mechanism to empower learners to become self-regulators of their own learning” based on the assumption that students would shift from surface to deep learning in the process of responding to the received feedback.

Background: active learning in Japanese higher education

The term AL has only recently been used in Japan. According to Mizokami (Citation2014), AL started attracting attention in Japanese higher education around 2010. He explains that the importance of AL was emphasised in the 2012 comprehensive report Qualitative Transformation of Undergraduate Education of the Central Council for Education and in the 2012–2015 national project entitled “Improving Higher Education for Industrial Needs” funded by the Ministry of Education, Culture, Sports, Science, and Technology. The use of AL in these high-level documents contributed to the spread of the term throughout Japanese higher education.

Definitions

Nagata and Hayashi (Citation2016) noted that two particular definitions of AL, formulated by the Central Council for Education (Citation2012) and Mizokami (Citation2014), are predominant in Japan. According to the former, unlike traditional lectures, AL, entailing the use of instructional methods, such as problem-based learning and cooperative learning may enable students to learn actively. Mizokami (Citation2014) defined AL as all types of learning other than passive learning (i.e. listening to lectures) that engage students in activities such as writing, speaking and presenting, entailing the externalisation of cognitive processes. In other words, student learning is demonstrated solely through activities such as speaking or writing.

Whereas some scholars and instructors endorse these definitions, others do not. For example, Hashimoto (Citation2017), who conducted a survey of representatives of national universities in Japan to elicit their understandings of AL, found that the respondents’ opinions varied. While some respondents reported that they agreed with one of the two prevailing AL definitions, others felt that a broader definition was required, arguing that even conventional lectures can be AL exercises, as long as students’ brains are activated towards learning.

Nakasono and Tanigawa (Citation2018) argued that the introduction of AL has led to confusion among instructors. They criticised the Central Council’s definition because it explicitly indicates that specific methods (e.g. problem-based learning and cooperative learning) should be used to foster AL without delineating associated contexts or the instructors’ roles. Consequently, instructors tend to regard AL as simply entailing the use of problem-based learning and cooperative learning methods even in the absence of instructional designs or interventions (Kojima, Citation2017). Nakasono and Tanigawa also questioned Mizokami’s definition, arguing that it does not allow for a consideration of learning as active unless externalised cognitive processes occur through particular activities such as writing and speaking. They are thus concerned that instructors may understand AL as merely employing these activities.

Both of the above-mentioned definitions foreground the use of specific methods or activities associated with AL. This is partly because unless students’ cognitive processes are externalised through these methods or activities, instructors would be unable to assess their learning (Nakai, Citation2015). Those who criticise these definitions have proposed a broader definition of AL as learning through engagement. Accordingly, any learning may be considered active as long as students are engaged in it. However, proponents of this broader definition have been unable to suggest any alternative approach for assessing AL outcomes. Even if students do not appear to be active, their minds may be active (and vice versa). Thus, both the broad and narrow definitions of AL have considerable limitations, and the process of further developing AL definitions has not advanced. It is therefore necessary to move beyond these restrictive definitions and adopt a different perspective to elucidate how AL should be understood. Accordingly, shifting the lens to focus on key actors in AL may further an understanding of AL perceptions and practices.

Nakasono and Tanigawa (Citation2018) have drawn attention to the impression conveyed by many publications on AL that students voluntarily or automatically become active learners through the use of instructional methods, such as problem-based learning and cooperative. They have pointed out that while the authors of these publications view students as the central agents, they do not engage in a comprehensive discussion of the instructors’ roles in AL. This is a critical gap, as the successful application AL is highly contingent on the understandings and actions of the instructors (Oyama & Taguchi, Citation2013). At the same time, the Central Council of Education (Citation2012) noted that AL is a two-way pedagogy in which instructors encourage students to engage in learning while students themselves engage in learning through their interactions with their instructors and peers. However, this conception of AL as a two-way pedagogy does not seem to be widely shared by instructors and scholars, as suggested by Nakasono and Tanigawa. Considering this context, our first research question was framed as follows: Do instructors understand AL approaches to be student-oriented (students voluntarily engage in learning on their own), instructor-oriented (instructors encourage students to become actively engaged in learning), or a two-way pedagogy (students and instructors engage in learning and teaching through their interactions with each other)? This question is pertinent as instructors’ understandings of AL fundamentally influence their teaching approaches, which in turn influence students’ learning approaches (Kember, Citation2009).

Practice

Given that the adoption of AL by all Japanese educational institutions has been strongly encouraged by the Ministry of Education, Culture, Sports, Science, and Technology, its implementation has become unavoidable (Mizokami, Citation2014). The number of faculty development workshops focusing on AL has dramatically increased in recent years (Mizokami, Citation2014). The implementation of AL often entails the application of problem-based learning, cooperative learning, and other types of class activities, as recommended by the Central Council of Education in its 2012 report (Aburatani, Kobayashi, Shimogoori, & Hongo, Citation2016). According to Iwasaki (Citation2015), problem-based leanring engages students in the learning process through their analysis and identification, of solutions that address the problems presented by the instructors. Students typically tackle these problems using a cooperative learning form. Some of the benefits of problem-based learning and cooperative learning include knowledge building, the development of generic skills, and increased motivation (Iwasaki, Citation2015). However, there is still uncertainty regarding the methods that Japanese instructors actually use in the classroom. Hence, we framed our second research question as follows: What methods do Japanese higher education instructions use to implement AL?

Assessment

Although AL is intended to encourage students to engage in learning activities by promoting learning that extends beyond mere knowledge acquisition, Matushita (Citation2016) observed that written exams are still the most common learning tool, even after the adoption of AL. However, the exclusive use of written exams for assessment implies that students will invest time and effort solely to prepare for these exams (Nakai, Citation2015). To address this issue, Matsushita (Citation2015) suggested that instructors could use performance-based assessment tools, such as reports and presentations. However, it remains unclear how Japanese university instructors actually assess students’ learning. Hence, we framed our third research question as follows: How do Japanese higher education instructors assess students learning outcomes using an AL approach?

Overall, the current study aims to answer the following research questions: Do instructors understand AL approaches to be student-oriented, instructor-oriented, or a two-way pedagogy? What methods do Japanese higher education instructions use to implement AL? How do Japanese higher education instructors assess students learning outcomes using an AL approach? In the following sections, the methodology is described followed by the results of the empirical study.

Methodology

Method

Questionnaire-based surveys entailing open-ended questions were employed to examine Japanese university instructors’ understandings and practices relating to AL by asking them how they understand the term AL and how they practice it. Open-ended questions are not only appropriate for exploratory studies such as this one (Beck, Czerniak, & Lumpe, Citation2000) but they also discourage irresponsible responses (e.g. the random selection of responses from multiple choices).

Sample

The surveys were administered by a professional research agency that maintains a database of over 10 million registered potential respondents recruited through online advertisements. The qualifications of these individuals are periodically reviewed and information on those who are disqualified (no longer teaching at universities) is removed. We requested the agency to select instructors teaching at the university level throughout Japan from this database and to ask them the specific questions indicated in this paper. The study was conducted in April 2019, and we received 412 completed questionnaires that were deemed valid.

Data analysis

Descriptive profiles based on general information about the respondents (e.g. sex and age) and their knowledge and use of AL were first compiled from the collected data. Content analysis was then performed on the data. CA classifies written texts into categories through coding for analysis (Cohen, Manion, & Morrison, Citation2011) and enables quantitative data analysis to be performed through the detection of frequencies and patterns that indicate significance (Vaismoradi, Jones, Turunen, & Snelgrove, Citation2016). In accordance with Erlingsson and Brysiewicz (Citation2017), the process of content analysis was documented as follows. The text data was first divided into smaller parts called “meaning units” that are any parts of the text that maintain meanings. In the current study, the responses to open-ended questions were divided into smaller meaning units, such as “AL is a learning style” and “AL is a teaching strategy.” These meaning units were labelled by formulating codes, usually one or two words long, that described each meaning unit the best; for example, “learning,” “teaching,” and “AL.” Categories are then formed by grouping the codes that are similar in their content or context (e.g. The respondent perceived AL as “Student-centred learning approach,” the respondent perceived AL as “Instructor-centred teaching approach” or the respondent perceived AL as “Two-way pedagogical approach”).

The analysed data were subsequently developed into illustrative categories of respondents’ understanding and practices relating to AL. Data on instructors’ understandings of AL were categorised as follows: (1) AL as student-centred, (2) AL as instructor-centred, (3) AL as a two-way pedagogy, (4) no definition of AL because of insufficient knowledge, and (5) other responses. Data relating to AL practices were divided into the following categories: (1) problem-based learning, (2) cooperative learning, (3) other class activities (e.g. presentations), (4) a lack of knowledge on how to adopt AL, (5) other responses, and (6)–(11) various combinations of some of the above categories. Lastly, data on assessments were divided into the following categories: (1) attitudes, (2) reports, (3) exams, (4) presentations, (5) mini tests, (6) other responses; and (7)–(25) combinations of the above categories.

Reliability and validity

To ensure inter-coder reliability, we respectively coded and categorised the data and established consensus (Saladaña, Citation2016). We independently reviewed the data and discussed any differences, to come to an agreement on categorisation to ensure reliability (Stemler, Citation2001). To minimise threats to the validity of research, we documented the processes applied to create the codes and categories. Following Silverman (Citation2017), we included deviant responses or direct quotes to strengthen the credibility of the data analysis and improve its validity. The NVivo (Version 10.0) software was used to organise and code the collected data.

Results

As shown in , nearly 90% of all respondents reported that they knew about AL. However, a little less than half of the respondents reported that they actually practiced it.

Table 1. General information about the respondents

Instructors’ understanding of active learning

As shown in , nearly 40% of the respondents who reported that they knew about AL defined it as a student-centred approach. One respondent stated that with AL, “students do research and presentations for themselves to acquire knowledge” (48–year-old male). Another respondent opined that “AL is a learning style that [enables] students [to] become active learners who actively participate in class activities, work on tasks and explore answers” (60-year-old male).

Table 2. Instructors’ understanding of active learning

While all of these responses entail reflections on certain aspects of AL, some of the instructors appeared to believe that AL is simply about letting students talk or do group work and learn by themselves without instructors’ substantial interventions. One respondent stated: “[In] AL, students play a central role, taking initiatives and control. However, there is a risk that students cannot reach any valid answers or [that they] come up with wrong answers without being corrected (57-year-old female).” This comment could imply a belief among instructors that they do not have to provide specific guidance to students when applying the AL approach.

Almost one-quarter of the respondents reported that AL is instructor-centred, with instructors assuming control over the process to facilitate students’ learning. One respondent stated: “AL is a teaching strategy in which instructors play a central role as facilitators to urge students to learn through presentations and debates” (44-year-old male). Another observed that “AL is a learning method in which instructors prepare students to actively learn. For example, we should encourage students to do research, presentations and group work” (56-year-old female).

Likewise, almost one-quarter of the respondents believed that AL is a two-way pedagogy. According to one respondent, “In AL, [it] is not that instructors teach one-way; both instructors and students work together to enable students to understand the course content more deeply” (69-year-old male). Another respondent explained that “in AL, both instructors and students interact each other to exchange opinions” (41-year-old male).

Twelve respondents stated that while they had reported that they knew about AL, they did not have sufficient knowledge to respond to the question on how they defined it. One said, “I don’t know the exact definition of AL” (45-year-old male). Another said, “I only know the term AL” (59-year-old male). The “other responses” included: “AL is about learning through field experiences” (41-year-old male) and “AL is learning that does not rely on writing” (68-year-old male).

To sum, although approximately 90% of the total respondents reported that they knew about AL, some of those who reported that they knew AL seem to have understood AL improperly (e.g. simply letting students do groupwork) or they did not have sufficient knowledge of AL to define it.

Instructors’ practices of active learning

, which shows AL practices, reveals that almost 75% of the respondents used problem-based learning, cooperative learning, or other classroom activities, or combinations of two or more of these activities. Apart from practices described in “other responses,” the most common practices reported by 35 respondents comprised a combination of problem-solving learning, cooperative learning and other class activities. Approximately 20 respondents reported using problem-based learning alone, cooperative learning alone, other class activities alone, a combination of problem-based learning and cooperative learning, or a combination of problem-based learning and other class activities. One of the respondents who reported the use of a combination of problem-based learning, cooperative learning and other classroom activities explained his practice as follows: “In class, I provide a theme and students do group work, [have] discussions and then make presentations about the theme in class” (40-year-old male). Another respondent made a similar statement: “I set a problem for students before a group discussion. Students first work individually on the problem, then [they] form a small group for discussions and finally [they] present their solutions to share [them with] the entire class” (46-year-old male). As discussed in the previous section and will also be discussed later, however, AL “requires more than simply incorporating more activities into the classroom” and needs to consider the context and the environment of the implementation (Kitchens, Means, & Tan, Citation2018, p. 59).

Table 3. Active learning practices

Six respondents reported that they did not know how to answer this question on methods used. One respondent stated: “It is hard to say which methods should be used in AL” (47-year-old male). Another respondent said, “I can’t think of anything particular to say” (37-year-old female). Examples from the “other responses” category included, “I ask students to write what they [have] learned on a piece of paper during the class” (64-year-old male) and “Q&A” (36-year-old male).

Assessment

shows the means by which instructors assessed students’ active learning outcomes. Attitudes were most commonly assessed, followed by reports, exams, combined attitudes and presentations, presentations and combined attitudes and exams.

Table 4. Assessment of students’ active learning outcomes

Attitudes

More than 20% of the respondents reported assessing AL outcomes based on students’ “attitudes” and 34 reported using attitudes in conjunction with other assessment tools. One respondent stated, “I look at whether students actively ask questions or make comments or listen to others. I evaluate students’ performance based on their attitudes” (62-year-old male). Another respondent explained that “since it is group work, I evaluate whether students actively participate in it or not. It does not matter whether what they said was correct or not” (55-year-old female).

Reports

Eighteen respondents stated that they used reports and 29 noted that they used reports in conjunction with other means of assessment. One respondent stated that “after the projects, students submit reports [through which] I evaluate students’ understanding of the content” (38-year-old female). Another said, “I make sure of students’ understanding of the learning content through reports” (50-year-old male).

Exams

Fourteen respondents reported that they assessed students through written exams while another 30 reported using exams in conjunction with other tools. One respondent said, “[while employing AL], I only evaluate students’ performance through exams” (46-year-old male). Another explained that “the exam includes questions that test knowledge application, so I just use exams for the evaluation” (42-year-old male).

Presentations

Nine respondents reported that they used presentations as an AL assessment tool while seven others reported its use in conjunction with other tools. One respondent explained that “when students do group presentations, I first give the same points to all of them within the same group. Then, I give bonus points to those who make extra contributions” (44-year-old female).

Mini tests

Only one respondent reported using a mini test for assessing AL outcomes while four respondents reported its use in conjunction with other tools. One of these respondents explained that “at the beginning of almost every session, I conduct a mini test for the students to assess how much of the learning content they have understood” (54-year-old male).

Others

Sixty-three responses that were categorised as “other responses” indicated the use of in-class essays and self-evaluation sheets for assessments.

In-class essays

Five respondents noted the use of in-class essays. Four others reported its use in conjunction with other tools. One respondent said that “after the class, students submit short essays about their learning” (52-year-old male). Another said, “I use reflective papers written by students for grading [purposes]” (63-year-old male).

Self-evaluations

Five respondents reported the use of student self-evaluation forms. Four others reported the use of self-evaluation forms in conjunction with other tools. One respondent said, “I look at whether students can analyse a problem from multiple perspectives. However, it is difficult to assess many students at once, so I rely on their self-evaluations” (61-year-old male). Another respondent similarly stated: “I give students self-evaluation sheets to measure their own learning progress” (27-year-old female).

No assessment

Some respondents reported that they do not assess AL because they find it difficult to do so. For instance, one respondent said, “It is impossible to evaluate AL. In essence, AL will be evident through its effects after students graduate from college” (43-year-old male). Another said, “I do not evaluate students [in terms of AL] because there are too many students to assess” (53-year-old male).

To sum, nearly three-quarters of respondents reported using problem-based learning, cooperative learning or other types of classroom activities. Also, over 20% of the respondents used students’ attitudes often without other means such as reports, exams, or presentations. It may be noteworthy, however, that none of the respondents reported quantifying his or her assessment on students’ attitudes.

Discussion

The findings of this study revealed that nearly 40% of the surveyed respondents reported using AL as a student-centred approach, 25% reported using AL as an instructor-centred approach, and a further 25% reported using AL as a two-way pedagogical approach entailing student-instructor interactions. While a consideration of AL as a student-centred approach is reasonable, further content analysis revealed that some instructors appeared to believe that they should simply let students do activities, such as problem-based or cooperative learning, on their own without their substantial interventions when applying AL (White et al., Citation2014). However, merely undertaking these activities is not equivalent to engagement in learning. This misinterpretation may be caused by faculty’s lack of familiarity’s with AL (Niemi, Citation2002) or a lack of management skills necessary to facilitate AL (Snyder, Citation2003).

Mizokami (Citation2014) argued that the emergence of AL signifies a paradigm shift from instructors’ lecture-based teaching to students’ activity-based learning. However, as McCullough and Munro (Citation2018) argue, AL and lectures should not be mutually exclusive because combining lectures with AL may help address the disadvantages of the latter (e.g. reduction in content coverage). As explained earlier (Chi & Wylie, Citation2014), engagement can be considered from multiple perspectives (from motivational to cognitive) at different modes of AL (from passive to interactive). That is, both activity-based learning and lecture-based learning can be learning with engagement. Yet, as noted by Matushita (Citation2016), the paradigmatic pendulum has now swung too far away from lecture-based teaching towards activity-based learning so that AL may have become too student-centred, as the current study suggests. It may be necessary to attempt to reverse this trend partially in the direction of lecture-based teaching to address the pedagogical imbalance. Promoting the concept of AL as a two-way pedagogy rather than as a student-centred approach may contribute to this goal (Cook-Sather, Citation2011). As Mok (Citation2006) explains, although active learning is often considered to be an approach where students take initiatives to be responsible for their learning, faculty are the ones who plan and designs lessons to facilitate students’ knowledge acquisition and construction. That is, “teachers and students share as partners the responsibilities and privileges of learning” (Barnes, Christensen, & Hansen, Citation1994).

When describing their AL-related practices, the majority of the respondents reported varying practices, notably the exclusive use of problem-based learning, cooperative learning, or other activities or combinations of two or more of these tools. These are arguably the most commonly used AL methods (Rico & Ertmer, Citation2015; Riley & Ward, 2016). It is noteworthy that of the various options, the combined use of all of these methods was the most popular response, perhaps being indicative of instructors’ efforts and struggles to implement AL. By contrast, approximately 75% of the respondents reported that they separately assessed attitudes, reports, exams and so forth (i.e. those who used attitudes only used attitudes) for assessing AL without attempting to triangulate or combine the use of these tools. While students’ attitudes are considered to be associated with their participation in group activities (Zou, Schunn, Wang, & Zhang, Citation2017), it may be striking that over 20% of the respondents reported exclusively using students’ attitudes to assess them, arguably depending solely on their observations or feelings without quantifying them.

The findings of this study appear to indicate that in general, quite a few Japanese university instructors may not understand AL properly, focusing on the use of instructional methods such as problem-based learning and cooperative learning and often using students’ attitudes to assess their learning outcomes without indicating how they measured these attitudes. That is, some instructors seem to equate these instructional methods with AL, even in the absence of knowledge regarding their mechanisms and ways of measuring learning outcomes. This situation is alarming and points to the necessity of developing an understanding of AL approaches and their application among instructors, wherein these approaches are situated within a broader pedagogical design or methodology that entails a theory of learning, including the aforesaid constructivism.

Conclusion

This study has explored understandings and practices of AL among Japanese university instructors, suggesting alternatives to the perceived equation of AL with instructional methods. Following the implementation of a series of educational policy reforms, the pedagogical paradigm shifted from instructor-centred teaching to student-centred learning. However, the pendulum has swung too far from lecture-based teaching to activity-based learning, causing confusion, among instructors, that AL means undertaking activities such as group discussions, while these activities do not necessarily engage students in learning. Indeed, when applying AL, faculty’s intervention is important because student engagement can be encouraged through both activity-based learning and lecture-based learning from multiple perspectives at different modes. To address this issue, the concept of AL should be promoted as a two-way approach to teaching and learning rather than solely as a student-centred approach to learning.

In light of this situation, the Central Council for Education (Citation2017) did not use the term AL in its 2017 curricular guidelines, pointing to difficulties relating to its definition and comprehension (Nishioka, Citation2017). Thus, it is possible that the Japanese government is now apprehensive that the superficial popularity of AL, without a well-founded understanding of the approach has created confusion. They may consequently decide to abandon it even before it has been properly established. However, we believe that AL’s greatest contribution to education, at least in the context of Japanese higher education, lies precisely in the creation of confusion among instructors and scholars alike, thereby leading to the broadening, deepening, and strengthening of arguments on teaching and learning. Although AL is a pedagogy or pedagogical methodology, it is defined as “learning” and not as “active teaching.” We argue that this is because the concept implies mutual learning between instructors and students through the active engagement of students in learning.

Limitations

The current study had several limitations that should be addressed in future studies. First, because our sample was recruited through a research agency, purely random sampling was not conducted. Consequently, sampling biases may have occurred. However, as respondents were sampled in all of the Japanese prefectures and covered a broad range of ages, the sample is likely to be representative of the general population. Second, given that the current study was exploratory in nature, future studies could engage in a deeper analysis of the findings. For example, some respondents may not have agreed with our categorisations of their understanding and practices relating to AL. Furthermore, there is a possibility that instructors actually used an AL-based teaching style without naming it as such, as reported by one respondent.

To address these limitations, survey questionnaires comprising multiple-choice questions could be administered for a randomly selected sample. Because these questions are pre-set and not open-ended (as in our survey), respondents simply have to select a response. The above limitations would thus be addressed through randomly collected data. However, to ensure validity, such quantitative data should be triangulated with the findings of qualitative studies, such as this one, which can be used to design future questionnaires. Future quantitative studies along with the findings of this qualitative study will undoubtedly contribute to the formulation of strategies for facilitating the adoption of the AL by shedding further light on instructors’ understandings and practices relating to AL.

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Notes on contributors

Hiroshi Ito

Dr. Hiroshi Ito is a professor at NUCB Business School, Nagoya University of Commerce and Business. He received his MA from Teachers College, Columbia University, PhD from the University of California, Los Angeles, MBA from the Université Paris I Pantheon-Sorbonne, and PhD from Nagoya University. Before coming to NUCB Business School, he had worked as an education expert for UNESCO (France), UNICEF (the Philippines), JICA (Ecuador and Paraguay), an NGO (Guatemala) and an educational service company (USA). His research interests lie in pedagogy and educational assessment as well as social marketing and environmental policy. He has published articles in the following: Compare: A Journal of Comparative and International Education, Urban Research and Practice, Applied Environmental Education & Communication, Policy Design and Practice, Journal of Further and Higher Education, and Journal of Nonprofit and Public Sector Marketing, as well as many others.

Shinichi Takeuchi

With dual expertise in education and business administration, Dr. Shinichi Takeuchi has extensive achievements in organisational implementation of case method in both academic and business settings. Prior to pursuing an academic career, Dr. Takeuchi was actively engaged in business administration and personnel training as well as in strategic reform and exchange projects in Japan and overseas at Mazda Motor Corporation. He is also the founder and representative director of Case Method Laboratory, Inc., which specialises in capacity development through case method. His academic background ranges from philosophy of education, educational methods, history of professional education, school administration and organisational behaviour to managerial accounting. He has authored a number of books, research papers and articles on case method education.

Notes

1. Instructors refer to higher education lecturers in this paper.

References

  • Aburatani, H., Kobayashi, J., Shimogoori, A., & Hongo, T. (2016). Jyugyouryoku appu: Akutibu raaningu [Effective active learning: Strategic essentials for discovering and developing your teaching skills] ( M. Ichitsubo, Ed.). Tokyo: Jikkyo Syuppan.
  • Akiva, T., Cortina, K. S., Eccles, J. S., & Smith, C. (2013). Youth belonging and cognitive engagement in organized activities: A large-scale field study. Journal of Applied Psychology, 34, 208–218.
  • Barkley, E. F. (2009). Student engagement techniques: A handbook for college faculty. San Francisco, CA: Jossey-Bass.
  • Barnes, L. B., Christensen, C. R., & Hansen, A. J. (1994). Teaching and the case method. Cambridge, MA: Harvard Business School Press.
  • Beck, J. C., Czerniak, M., & Lumpe, A. T. (2000). An exploratory study of teacher’s beliefs regarding the implementation of constructivism in their classrooms. Journal of Science Teacher Education, 11(4), 323–343.
  • Biase, R. D. (2019). Moving beyond the teacher-centered/learner-centered dichotomy: Implementing a structured model of active learning in the Maldives. COMPARE: A Journal of Comparative and International Education, 49(4), 565–583.
  • Carey, P. (2013). Student engagement: Stakeholder perspectives on course representation in university governance. Studies in Higher Education, 38, 1290–1304.
  • Central Council for Education. (2012). Qualitative transformation of undergraduate education. Retrieved from https://www.mext.go.jp/component/b_menu/shingi/toushin/__icsFiles/afieldfile/2012/10/04/1325048_1.pdf
  • Central Council for Education. (2017). Atarashii gakusyushidou youryou no kangaekata [Ways of thinking toward new curricular guidelines]. Retrieved from https://www.mext.go.jp/a_menu/shotou/new-cs/__icsFiles/afieldfile/2017/09/28/1396716_1.pdf
  • Chan, C. W. H., Leung, D. Y. P., Lee, D. T. F., Chair, S. Y., Ip, W. Y., & Sit, J. W. H. (2018). The perception of generic capabilities and learning environment among undergraduate nursing students after the implementation of a senior intake scheme. Journal of Further and Higher Education, 42(2), 158–165.
  • Chi, M. T. H., & Wylie, R. (2014). The ICAP framework: Linking cognitive engagement to active learning outcomes. Education Psychologist, 49(4), 219–243.
  • Cohen, L., Manion, L., & Morrison, K. (2011). Research methods in education. London: Routledge.
  • Cook-Sather, A. (2011). Lessons in higher education: Five pedagogical practices that promote active learning for faculty and students. Journal of Faculty Development, 25(3), 33–39.
  • Dadach, Z. E. (2013). First quantitative measurement of motivation. Study of the effects of active learning strategies. Norderstedt: GRIN.
  • Dancer, D., & Kamvounias, P. (2005). Student involvement in assessment: A project designed to assess class participation fairly and reliably. Assessment & Evaluation in Higher Education, 30, 445–454.
  • Deeley, S. J. (2014). Summative co-assessment: A deep learning approach to enhancing employability skills and attributes. Active Learning in Active Learning, 15(1), 39–51.
  • Diamond, N. S., Koernig, K., & Iqbal, Z. (2008). Uniting active and deep learning to teach problem-solving skills. Journal of Marketing Education, 30(2), 116–129.
  • Drew, V., & Mackie, L. (2011). Extending the constructs of active learning: Implications for teachers’ pedagogy and practice. The Curriculum Journal, 22(4), 451–467.
  • Erlingsson, C., & Brysiewicz, P. (2017). A hands-on guide to doing content analysis. African Journal of Emergency Medicine, 7(3), 93–99.
  • Esterhazy, R., & Damşa, C. (2017). Unpacking the feedback process: An analysis of undergraduate students’ interactional mean-making of feedback comments. Studies in Higher Education, 44(2), 260–274.
  • Freeman, S., Eddy, S. L., McDonough, M., Smith, M. K., Okoroafor, N., Jordt, H., & Wenderoth, M. P. (2014). Active learning increases student performance in science, engineering, and mathematics. PNAS, 111(23), 8410–8415.
  • Gibbs, R., & Poskitt, J. (2010). Student engagement in the middle years of schooling (years 7–10): A literature review. Wellington: Ministry of Education.
  • Golightly, A., & Raath, S. (2015). Problem-based learning to foster deep learning in preservice geography teacher education. Journal of Geography, 114, 58–68.
  • Graeff, T. R. (2010). Strategic teaching for active learning. Marketing Education Review, 20(3), 265–278.
  • Grosas, A. B., Raju, S. R., Schuett, B. S., Chuck, J.-A., & Miller, T. J. (2016). Determining if active learning through a formative assessment process translates to better performance in summative assessment. Studies in Higher Education, 41(9), 1595–1611.
  • Harrison, R. (2010). Learner managed learning: Managing to learn or learning to manage? International Journal of Lifelong Education, 19, 312–321.
  • Hashimoto, M. (2017). Raito actibu raaningu no susume [Encouragements of light active learning]. Tokyo: Nakanishiya Syuppan.
  • Hmelo-Silver, C. E. (2004). Problem-based learning: What and how do students learn? Educational Psychology Review, 16(3), 235–266.
  • Hurd, P. D. (2000). Active learning. Journal of Pharmacy Teaching, 7(3/4), 29–47.
  • Hussain, R. M. R., Mamat, W. H. W., Salleh, N., Saat, R. M., & Harland, T. (2007). Problem-based learning in Asian universities. Studies in Higher Education, 32(6), 761–772.
  • Ito, H. (2017). Rethinking active learning in the context of Japanese higher education. Cogent Education, 4(1), 1–10.
  • Iwasaki, C. (2015). Daigakusei no manabi wo hagukumu gakusyu kankyou no dezain: Atarashii paradaimu ga hiraku akutibu raaningu [Designs for learning environments that enhance undergraduate learning: Challenges toward active learning led by the new paradigm]. Osaka, Japan: Kansai University Press.
  • Kakuhou, M., Matsumura, N., & Hirata, F. (2011). Syuugyouryoku to daigaku kaikaku [Employability skills and the higher education reform]. Tokyo: Gakuji Syuppan.
  • Kawano, A. (2016). Kyouin yousei katei ni okeru akutibu raaningu no kadai to tenbou [The review of the enactment of active learning for pre-service teacher education]. Bulletin of Joetsu University Education, 35, 43–55.
  • Kember, D. (2009). Promoting student-centered forms of learning across an entire university. Higher Education, 58, 1–13.
  • Kember, D., & Leung, D. Y. P. (2005). The influence of active learning experiences on the development of graduate capabilities. Studies in Higher Education, 30(2), 155–170.
  • Kinoshita, T. J., Knight, D. B., & Gibbes, B. (2017). The positive influence of active learning in a lecture hall: An analysis of normalised gain scores in introductory environmental engineering. Innovations in Education and Teaching International, 54(3), 275–284.
  • Kitchens, B., Means, T., & Tan, Y. (2018). Captivate: Building blocks for implementing active learning. Journal of Education for Business, 93(2), 58–73.
  • Kojima, M. (2017). ‘Saiko akutibu raaningu’ o yonde [Reading ‘rethinking of active learning’]. Medical Education, 48(5), 304.
  • Laverie, D. A. (2006). In-class active cooperative learning: A way to build knowledge and skills in marketing courses. Marketing Education Review, 16(2), 59–76.
  • Lee, J.-S. (2014). The relationship between student engagement and academic performance: Is it a myth or reality? The Journal of Educational Research, 107, 177–185.
  • Levant, Y., Coulmont, M., & Sandu, R. (2016). Business simulation as an active learning activity for developing soft skills. Accounting Education, 25(4), 368–395.
  • Matsushita, K. (2015). Diipu akutibu raaningu e no izanai [Invitation to deep active learning]. In K. Matsushita (Ed.), Deep active learning: Deepening higher learning (pp. 1–27). Tokyo: Keiso-Shobo.
  • Matushita, K. (2016). Akutibu raaningu o dou hyouka suruka [How to assess active learning]. In K. Matsushita & T. Ishii (Eds.), Akutibu raaningu no hyouka [Assessment of active learning]. Tokyo: Toshindo. pp.3-25.
  • McCullough, K., & Munro, N. (2018). Finance students’ experiences of lecture-based active learning tasks. Innovations in Education and Teaching International, 55(1), 65–73.
  • Michael, J. (2007). Faculty perceptions about barriers to active learning. College Teaching, 55(2), 42–47.
  • Ministry of Education, Culture, Science, Sports & Technology. (2012). Sangyoukai no niizu ni taioushita kyouikukaizen jyuujitutaiseiseibijigyou [Improving higher education for industrial needs]. Retrieved from https://www.mext.go.jp/a_menu/koutou/kaikaku/sangyou/
  • Misseyanni, A., Lytras, M. D., Papadopoulou, P., & Marouli, C. (2018). Active learning strategies in higher education: Teaching for leadership, innovation, and creativity. Bingley: Emerald.
  • Mizokami, S. (2014). Akutibu raaningu to kyoujyugakusyuu parademu no tankan [Active learning and teaching/learning paradigm shift]. Tokyo: Toshin-do.
  • Mok, I. A. C. (2006). Teacher-dominating lessons in Shanghai: The insiders’ story: Mathematics classrooms in twelve countries. Leiden: Brill Sense.
  • Nagata, T., & Hayashi, K. (2016). Akutibu raaningu no dezain [Practical designs for active learning]. Tokyo: Tokyo University Press.
  • Nakai, T. (2015). Akutibu raaningu [Active learning]. Tokyo: Tamagawa University Press.
  • Nakasono, A., & Tanigawa, H. (2018). Akutibu Raaningu saikou [Rethinking active learning]. In Akutibu raaningu hihanteki nyuumon [Practical guides to ‘really’ active learning and its critical re-evaluation]. Tokyo: Nakanishiya Publishing. pp. 11-21.
  • Ní Raghallaigh, M., & Cunniffe, R. (2013). Creating a safe climate for active learning and student engagement: An example from an introductory social work module. Teaching in Higher Education, 18(1), 93–105.
  • Niemi, H. (2002). Active learning – A cultural change needed in teacher education and schools. Teacher and Teacher Education, 18(7), 763–780.
  • Nishioka, K. (2017). Nihibei ni okeru akutibu raaninguron no seiritsu to tenkai [Establishment and development of active learning in Japan and the USA]. Journal of Education, 84(3), 25–32.
  • O’Grady, A., Simmie, G. M., & Kennedy, T. (2014). Why change to active learning? Pre-service and in-service science teachers’ perceptions. European Journal of Teacher Education, 37(1), 35–50.
  • Ogata, N., de Weert, E., & Yoshimoto, K. (2007). Policies on the transition from higher education to employment since the 1990s. In J. Allen, Y. Inenaga, R. van der Velden, & K. Yoshimoto (Eds.), Competencies, higher education and career in Japan and the Netherlands (pp. 25–50). Dordrecht, The Netherlands: Springer.
  • Oyama, M., & Taguchi, M. (2013). Daigaku ni okeru guruupu gakusyu no ruikeika [Categorization of group learning in higher education]. Journal of Educational Technology in Japan, 37(2), 129–143.
  • Peters, R. A. (2011). Enhancing academic achievement by identifying and minimizing the impediments to active learning. Public Administration Quarterly, 35(4), 466–493.
  • Revell, A., & Wainwight, E. (2009). What makes lectures ‘Unmissable’? Insights into teaching excellence and active learning. Journal of Geography in Higher Education, 33(2), 209–223.
  • Rico, R., & Ertmer, P. A. (2015). Examining the role of the instructor in problem-centered instruction. TechTrends, 59(4), 96–103.
  • Riley, J., & Ward, K. (2017). Active learning, cooperative active learning, and passive learning methods in an accounting information systems course. Issues in Accounting Education, 32(2), 1–16.
  • Rushton, A. (2005). Formative assessment: A key to deep learning? Medical Teacher, 27(6), 509–513.
  • Saladaña, J. (2016). The coding manual for qualitative researchers. Thousand Oaks, California: SAGE.
  • Sedaghat, M., Abedin, A., Hejazi, E., & Hassanabadi, H. (2011). Motivation, cognitive engagement, and academic achievement. Procedia – Social and Behavioral Sciences, 15, 2406–2410.
  • Silverman, D. (2017). Doing qualitative research. Thousand Oaks, CA: SAGE Publications.
  • Smiley, W., & Anderson, R. (2011). Measuring students’ cognitive engagement on assessment tests: A confirmatory factor analysis of the short form of the cognitive engagement scale. Research and Practice in Assessment, 6, 17–28.
  • Snyder, K. D. (2003). Ropes, poles, and space: Active learning in business education. Active Learning in Higher Education, 4(2), 159–167.
  • Stemler, S. (2001). An over view of content analysis. Practical Assessment, Research & Evaluation, 7(17), 1–10.
  • Tangney, S. (2014). Student-centered learning: A humanist perspective. Teaching in Higher Education, 19(3), 266–275.
  • Taylor, J. E., & Ku, H.-Y. (2011). Measuring active learning to predict course quality. Performance Improvement Quarterly, 24(1), 31–48.
  • Vaismoradi, M., Jones, J., Turunen, H., & Snelgrove, S. (2016). Theme development in qualitative content analysis and thematic analysis. Journal of Nursing Education and Practice, 6(5), 100–108.
  • Virtanen, A., & Tynjälä, P. (2019). Factors explaining the learning of generic skills: A study of university students’ experiences. Teaching in Higher Education, 24(7), 880–894.
  • Walker, C. O., Greene, B. A., & Mansell, R. A. (2006). Identification with academics, intrinsic/extrinsic motivation, and self-efficacy as predictors of cognitive engagement. Learning and Individual Differences, 16(1), 1–12.
  • White, C., Bradley, E., Martindale, J., Roy, P., Patel, K., Yoon, M., & Worden, M. K. (2014). Why are medical students ‘checking out’ of active learning in a new curriculum? Medical Teacher, 48, 315–324.
  • Wilson, B. M., Pollock, P. H., & Hamann, K. (2007). Does active learning enhance learner outcomes? Evidence from discussion participation in online classes. Journal of Political Science Education, 3, 131–142.
  • Wimpenny, K., & Savin-Baden, M. (2013). Alienation, agency and authenticity: A synthesis of the literature of student engagement. Teaching in Higher Education, 18, 311–326.
  • Zou, Y., Schunn, C. D., Wang, Y., & Zhang, F. (2017). Student attitudes that predict participation in peer assessment. Assessment & Evaluation in Higher Education, 43(5), 800–811.