Conceptualizing rigor and its implications for education in the era of the Common Core

Abstract

The adoption of Common Core State Standards in the USA by 46 states and the District of Columbia has provided several new foci for K-12 instruction, not the least of which is the reading and understanding of complex text, a higher order thinking process. Closely associated with this is the notion of rigor, the focus of the present study. As educators who work with administrators and teachers across the country, we have noticed that while there exists a general concern about rigor, there is not a coherent understanding of what it is. As such, there is a need to establish a common understanding of rigor that is useful for school-based personnel. Additionally, we propose that it’s important for educators to regularly measure and track rigor as one part of a larger dashboard of quality indicators which can inform school leaders on the educational health of their school.

Keywords:

• rigor
• higher order thinking
• knowledge work
• engagement
• Webb’s DOK

Public Interest Statement

For many decades, the notion of a rigorous class has been associated with demanding classes, difficult tests, and abundant homework requiring significant student work, often in the form of memorization. In the age of the Common Core State Standards, this traditional notion of rigor is now insufficient. We present the idea that rigor is better thought of as a continuum of cognitive simplicity–complexity embedded within the learning activities prepared for students by teachers. This puts the teacher directly in charge of manipulating rigor to encourage and develop cognitively complex thinking in students. We present empirical evidence that is part of over 16,000 classroom observations using Webb’s depth-of-knowledge continuum to measure the degree to which rigor is present in K-12 classrooms. Results show that instruction most often occurs at the simple end of the cognitive complexity continuum with little focus on higher order thinking that falls short of Common Core standards.

1. Introduction

In our work with administrators and teachers, we’ve found the topic of rigor to be increasingly popular; however, we’ve also noticed an interesting paradox. While there is almost unanimous agreement that rigor is important, a concise, universal definition is absent, thus leading directly to a second problem. What is the place of rigor within the educational system? And thirdly, not knowing what rigor is makes it impossible for school leaders to determine the extent to which it exists within instruction that for the large majority of states is implemented within the Common Core (2010) standards. As such, there exists a need within education to identify and agree on what constitutes rigor. In this article, we address three questions regarding rigor. First, how can rigor be defined in such a way as to be useful to teachers and educators in the age of the Common Core? Secondly, what is the place of rigor in education? And third, to what extent does the rigor of classroom instruction align with the expectations of the Common Core? To the first question, we explore long-held and current ideas about rigor, and then propose a broader conceptualization of the construct. We then suggest a model proposing the role rigor might occupy within the larger purpose of education. To the third question, we compare the rigor expectations of the Common Core for seventh-grade standards in literacy and mathematics to classroom data we have collected over the past three years. We conclude with thoughts on implications for rigor and instruction.

2. What is rigor?

Dictionaries define rigor as possessing the quality of severity or strictness, of being demanding and difficult. In past decades, the notion of educational rigor was likely to equate to the teacher whose class was “hard” as defined by plentiful homework, difficult tests, and only a few students being awarded an “A.” Within this traditional notion, rigor was used as a sorting mechanism to distinguish students who best learned within a traditional system of education, a notion which still exists in many schools today, particularly those espousing a “traditional” curriculum. For a principal, rigor might mean quiet hallways during class time suggesting students who are engaged and working hard. Rigor might also be reflected by few office referrals as teachers keep students “in line” and on task with “strict” class management skills. While these ideas may have defined rigor in the past and may still be alive and well in some schools, this traditional construct is insufficient in today’s educational context as it raises at least two critical questions. First, a traditional notion of rigor implies a dichotomous state where it is either consistently present or not, with no allowance for middle ground. Secondly, the construct is disconnected from the important issue of cognition, or thinking, which is inherent and fundamental to learning. As few would disagree that rigor is unimportant to schooling, we propose the need for a more productive construct of rigor that is more closely aligned to teaching and learning in the age of the Common Core.

To help with an updated conceptualization of rigor, we can turn to insights from Wagner (2008) who described his conversations with business leaders who shared their thoughts on what students must do to effectively compete in the twenty-first century. Wagner condensed his findings to seven essential skills which he equates to rigor. Two of these skills pertinent to our inquiry include the ability to access and analyze information, and secondly, to be able to think critically and solve problems. As a whole, Wagner’s skills are arguably outcomes that he suggests should occur as a result of exposure to a rigorous education. Additionally, both outcomes suggest cognitive expectations.

Strong, Silver, and Perini (2001) suggest that rigor can be thought of as a goal where the teacher strives to help students develop the capacity to understand content. Insights into a much more detailed definition of rigor may be inferred from a Delphi study conducted by the American Philosophical Association (Fancione, 1990, 2013). In addition to a number of personal traits including persistence, inquisitiveness, use of reason, and the ability to order complex matters, the purposeful use of evaluation, analysis, interpretation, and inference in thinking was also reported. While this definition captures many facets aligned to human thinking (cognition), several of which overlap with those of Wagner (2008), we think teachers and administrators would find it exceedingly difficult to apply them to the classroom.

Additionally, we disagree that rigor should be interpreted as a goal which is either achieved or not. Such interpretations turn rigor into a dichotomous phenomenon (it either exists or it does not) that is incongruent with the broad dimensionality of human cognition that is intrinsic to thinking, learning, and education. Rather, we agree with the statement by Thorndike when he commented that “Meaning … whatever exists at all exists in some amount” (1918, p. 16). The idea of “some amount” is what we see as critical to a conceptualization of rigor which we view as situated along a continuum of cognitive complexity.

3. Cognitive complexity

The theory of cognitive complexity, originally hypothesized to explain how individuals perceive their social world, provides a theory for thinking that we suggest is applicable when discussing the construct of rigor (Bieri, 1955; Kelly, 1955). The theory assumes that human behavior seeks greater predictability of the environment, and that every individual develops a system of constructs which enable such predictions. Thus, a collection of constructs compose the constraint-satisfaction system that is continually tested by the individual who makes predictions based upon the success or failure of the system. It is also assumed that when predictions are unsuccessful, the individual makes substantive changes to the constructs undergirding the system which are left unchanged when predictions are successful. Ultimately, the degree to which the individual’s system of constructs is successful reflects its cognitive simplicity–complexity (Bieri, 1955). A system that is successfully predictive is more complex, that is, it includes a greater number of more finely tuned constructs which provide greater constraint-satisfaction ability, and ultimately, greater predictive ability.

Now let’s discard the notion that rigor is either absent or present, and explore the idea that it exists in some amount. Consider the example of a thermometer, which of course measures the presence of heat. We can likely agree that on a hot summer day, heat exists to a greater extent than on a cold and blustery winter day. At the same time, it would be incorrect to say there is no heat on the cold day, rather there is just less of it than on the hot day. In this sense, heat is not an alternative of either “hot” or none at all, instead it is a question of how much is present. We propose this same state can be applied to rigor. If we let go of the idea that rigor is a dichotomous variable that is either non-existent or exists in great quantity, we become conceptually unencumbered to envision an alternate paradigm. In such a paradigm, we conceptualize that rigor applied to instruction, like temperature, is always present along a continuum of cognitive simplicity–complexity. So, the question no longer addresses whether rigor is present or not, rather the question becomes to what degree is it present? This conceptualization allows for an interpretation of rigor that accounts for the many gradations of cognitive simplicity–complexity in thinking, and reflects other current thinking, such as that found in the rigor/relevance framework (Daggett, 2000, 2014).

A primary implication for our paradigm is that rigor is now viewed as an independent variable that can be intentionally manipulated by the teacher within the range of learning activities that Schlechty (2005) calls knowledge work. These activities, designed by the teacher, can range from lecture, to reading or problem solving, to writing, group work, or class discussions, to presenting in front of others, to problem-based learning, to working math computations, and innumerable other activities. This makes knowledge work the essential delivery vehicle used by teachers to promote the desired cognitive simplicity–complexity of thinking in learning activities, and hence, is where rigor resides. We can now offer a definition of rigor that intertwines the components of cognitive simplicity–complexity, knowledge work, and existence on a continuum. As such, rigor is “the extent to which the knowledge work engaged in by students requires cognitive simplicity–complexity.”

This definition addresses at least four dimensions regarding rigor. First, rigor is now conceived as a continuum of cognition (thinking) ranging from simple to complex. Secondly, it allows for the quantification of rigor in the classroom and across a school building, which we address shortly. Thirdly, it recognizes that knowledge work at the lower end of the continuum is important because it creates the fundamental, cognitively simple knowledge essential to more cognitively complex thinking. And fourth, rigor is intentionally manipulated by the teacher to encourage a range of cognitively simple to complex thinking in students. The question now becomes how to determine the presence of rigor in instruction?

3.1. Cognitive taxonomies

Anderson and Krathwohl (2001) suggest that thinking is basically one of two processes, either recall or transfer. Recall involves the learning of fundamental information, while transfer of learning builds deeper insights which extend the individual’s schema, thus adding usefulness and meaning to recalled information. Transfer is where cognitive complexity in thinking is developed and tested for constraint-satisfaction by the individual, and where thinking grows from simple to increasingly complex. While there are several taxonomies for evaluating complexity of thinking, two of the most popular are from Bloom, Englehart, Furst, Hill, and Krathwohl (1956), including the updated version by Anderson et al. (2001). A second taxonomy has been developed by Webb (1997). In our experience working with hundreds of educators, we’ve found the six levels composing both Bloom’s and Anderson et al.’s taxonomies are often too fine grained and difficult for teachers to consistently identify. On the other hand, the four levels that make up Webb’s Depth of Knowledge (DOK) scale are much easier to observe and differentiate in practice (1 = recall; 2 = application; 3 = strategic thinking; and 4 = creating new knowledge). Applying the dichotomy offered by Anderson and Krathwohl (2001) to Webb’s DOK, the processes of application, strategic thinking, and the creation of new knowledge would be considered transfer activities signifying increasing cognitive complexity. Earlier we suggested that through the knowledge work they create, skilled teachers can intentionally manipulate the thinking level required of students and thus provide a range of knowledge work extending from cognitively simple to complex. Webb’s DOK provides an ordinal scale that assumes a continuous underlying distribution to assist teachers in the rating of these thinking activities. We use Webb’s DOK as a quantifiable proxy suggesting cognitive simplicity–complexity.

4. The role of rigor in education

To truly understand the role of rigor, it is insufficient to view it apart from the system of education, rather it must be viewed within the larger educational process. To help with this idea, Figure 1 provides a schematic of “If–Then” statements which detail the logic of where and how rigor integrates within the purpose of education.

Figure 1. The role of rigor in the educational process.

Our view of instruction is that of a complex system consisting of multiple components within which teachers design and implement lessons. Some of these components include the standard being taught, the pedagogy that will be utilized, the level of cognitive complexity at which students will be asked to think, the various materials which will be required, how time will be used, how the lesson will be differentiated, how formative and summative assessments will inform teaching and student progress, and appropriate class management considerations. Up to this point the lesson is still on paper. As we move from the design to the implementation or instructional delivery phase, each of these same components must be considered. Of course, it is during implementation where the design may be modified to fit particular circumstances that arise due to student reaction to the lesson. Regarding rigor, it is in the implementation phase where the thinking levels that were planned in the design phase must be assessed to determine whether or not they actually occurred as planned. In other words, did the intended complexity of thinking prepared in the lesson design occur during implementation, and if not, then why not?

Our model in Figure 1 is designed as reflexive in recognition of two considerations. First, educational stakeholders, traditionally at the community level, have the prerogative to change the goal(s) for education. Secondly, goals most often require continuous work and improvement. This makes their achievement an iterative pursuit that requires assessment of current efforts, evaluation of those efforts, and amendments to implementation based on result analysis. A reflexive model recognizes this continuous improvement cycle.

Let’s now consider the impact when cognitive complexity, or rigor, is an essential characteristic of the learning process. Our model reflects that if a rigorous education did in fact occur, then one outcome should be a student who is better able to utilize critical thinking skills, better able to analyze information and propositions, to integrate new information into current thinking, to evaluate alternatives, and to modify and improve existing processes or even create new ones. If education is responsible for training students to become cognitively complex thinkers, then the greater goal of an educated population that can better implement the responsibilities of citizenship is more likely to be achieved. While this is no simple or easily obtained outcome, and the process to create such thinkers involves complicated and difficult work that takes years of consistent effort by focused and professional teachers, a citizenry equipped to think with complexity is essential to the success of our country.

We now turn our attention to our final question. What is the state of rigor in regard to the Common Core?

5. The present state of rigor

Over the past three years, we’ve trained over 500 teachers and administrators to measure the presence of rigor in the classroom following an instructional rounds approach. The process requires just 1–2 minutes per classroom observation and has resulted in the gathering of 16,350 classroom observations across first through twelfth grade. The extent to which rigor exists is measured by evaluating the cognitive simplicity–complexity of student thinking using Webb’s DOK.

To record DOK, the observer enters the classroom and, after observing the knowledge work engaged in by students, records the applicable DOK level (1, 2, 3, or 4) for the class. The observer also records the grade level and subject being taught before moving on to the next classroom. Notice that the observation is only concerned with students and that no data are gathered regarding the teacher.

While it is not our intent to provide a detailed methodological description associated with the collection of the data in this discussion, we do think it’s important to provide evidence of inter-rater reliability in the observation of DOK. In a previously published study, we reported inter-rater reliability for the observation of DOK resulting in a Cohen’s kappa equal to 0.73 (Paige, Sizemore, & Neace, 2013). To investigate the consistency of the DOK measures in the complete data-set, a split-halves strategy was used. A random sample of 9,043 of the 16,350 total measures was selected and assigned a code of 0. The remaining 8,964 cases were assigned a code of 1. A chi-square test of independence for the two halves of the sample revealed no significant differences (χ² = 2.70, df = 3, p = 0.440). The results in Table 1 indicate the two halves are independent and the DOK measures are consistent across the entire sample.

Table 1. Results of DOK by split-half cross-tabulation for inter-rater reliability

			Split file		Total
			Not selected	Selected
DOK	Recall	Count	3,944	3,796	7,740
		Within split (%)	47.9	46.7	47.3
	Application	Count	3,260	3,272	6,532
		Within split (%)	39.6	40.3	40.0
	Critical thinking	Count	792	809	1,601
		Within split (%)	9.6	10.0	9.8
	Create	Count	232	245	477
		Within split (%)	2.8	3.0	2.9
Total		Count	8,228	8,122	16,350
		Within split (%)	100.0	100.0	100.0

To determine the extent to which cognitive complexity as outlined in the Common Core manifests itself in classroom instruction, we provide an analysis of seventh-grade English Language Arts (ELA) (n = 1,880 observations) and math (n = 1,471 observations) classrooms. We chose seventh grade as an example because, first, subjects in this grade are taught by content experts with deep knowledge of the content, enabling the ability to design lessons across cognitive simplicity–complexity. Secondly, content at the seventh-grade level easily lends itself to a range of cognitive simple–complex learning activities.

To determine the DOK target levels of seventh-grade ELA and math standards pursuant to the Common Core, we consulted a study by Sato, Lagunoff, and Worth (2011). The authors analyzed all applicable standards and assigned either a specific DOK level to each or, in most cases, assigned a range of two to three DOK levels. This provides a distribution of target DOK levels which can be compared to observed levels.

Table 2 organizes 39 Common Core State Standards (CCSS) seventh-grade ELA standards by DOK. The table reveals that 20 standards equate with learning at the DOK1 level (17.1%), while 35 and 42 standards lend themselves to DOK2 (29.9%) and DOK3 (34.2%) instruction, respectively. Finally, 22 standards connect to DOK4 (18.8%). Table 3 shows the distribution of observed DOK levels where 40.8% occurred at DOK1, 41.8% at DOK2, 13.8% at DOK3, and 3.6% at DOK4. Table 4 compares the DOK levels observed in the classroom to the CCSS target levels. At DOK1 and DOK2, observed instances far exceeded those at the CCSS target level, while at the DOK3 and DOK4 levels the relationship was reversed. Figure 2 graphs the distribution frequency of observed versus target CCSS DOK levels and shows a clear bifurcation between the two where the observed distribution is below that of the target distribution.

Table 2. DOK levels for seventh-grade ELA Common Core Standards

DOK level	n	%
1. Recall	20	17.1
2. Application	35	29.9
3. Critical thinking	42	34.2
4. Create	22	18.8
Total	119	100.0

Table 3. Observations for seventh-grade ELA by DOK

DOK level	n	%
1. Recall	767	40.8
2. Application	785	41.8
3. Critical thinking	260	13.8
4. Create	68	3.6
Total	1,880	100.0

Table 4. Comparison of observed and CCSS ELA standards by DOK

DOK level	Observed %	CCSS %	Difference
1. Recall	40.8	17.1	23.7
2. Application	41.8	29.9	11.9
3. Critical thinking	13.8	34.2	−(20.4)
4. Create	3.6	18.8	−(15.2)

Figure 2. Percent distribution of seventh-grade ELA observed and CCSS target DOK levels.

Table 5 shows the distribution of CCSS targets by DOK for seventh-grade mathematics where 37.5% are DOK1, 45.8% are DOK2, 16.7% are DOK3, and none are DOK4. Observed DOK levels are shown in Table 6 and reveal that 35.3% occurred at DOK1, 52.3% at DOK2, 11.4% at DOK3, and 1% at DOK4. Finally, Table 7 compares the two distributions and shows only slight differences between target and observed DOK levels which is further illuminated in Figure 3.

Table 5. DOK levels for CCSS seventh-grade math

DOK level	n	%
1. Recall	18	37.5
2. Application	22	45.8
3. Critical thinking	8	16.7
4. Create	0	0.0
Total	48	100.0

Table 6. Number of observations by DOK level for math

DOK level	n	%
1. Recall	520	35.3
2. Application	770	52.3
3. Critical thinking	168	11.4
4. Create	15	1.0
Total	1,473	100.0

Table 7. Comparison of observed and CCSS math standards by DOK

DOK level	Observed %	CCSS %	Difference
1. Recall	35.3	37.5	(2.2)
2. Application	52.3	45.8	(6.5)
3. Critical thinking	11.4	16.7	(5.3)
4. Create	1.0	0.0	1.0

Figure 3. Percent distribution of seventh-grade math observed and CCSS target DOK levels.

In summary, the observed rigor distributions for seventh-grade ELA falls considerably short of target CCSS levels, while the two distributions for math are strikingly similar. We do not think that the math teachers observed across 1,473 observations are intentionally planning instruction with rigor in mind. Rather, we speculate that the fundamental process of teaching mathematics through the direct instruction model where explanation and teacher modeling, a DOK1 activity, is typically followed by students solving problems, a DOK2 activity is responsible for the observed DOK distributions. The observed data suggest an approximate 3:5 ratio of DOK1:DOK2 instruction with the remaining 11% or so at the DOK3 level. CSSS target DOK levels for ELA show an almost even distribution between DOK1 and DOK2 instruction with the remaining 17.4% dedicated to DOK3 and DOK4 activities. The 1,880 ELA observations graphed in Figure 2 suggest that teachers implement little instruction emphasizing cognitive complexity.

6. Concluding thoughts

Our findings raise a central question regarding the general level of rigor which quickly leads to a discussion of how teachers are trained and how they in turn, train students to think. We ask you to consider for a moment the monumental shift in accountability standards from No Child Left Behind (NCLB), to those in the Common Core. Due to NCLB, teachers have been preparing students for well over a decade to think primarily at the recall and application levels due largely to end-of-year assessments emphasizing thinking at the cognitively simple end of the rigor continuum. An analysis of the CCSS rigor levels show that 53 and 16.7% of ELA and math standards respectively, can be taught at the DOK3/4 level. However, our observations reveal just 17.4% of ELA and 12.4% of math observations occurred at these levels. We think this strongly suggests that a shift must occur in how students are trained to think if they will be prepared to respond successfully to Common Core assessments emphasizing strategic thinking and emerge from the educational process as critical thinkers.

This leads us to the proverbial crossroads, the intersection where the knowledge work prepared by teachers meets with how we train students to think, and so we conclude with these two questions. First, do you know the extent to which rigor exists in your classroom, school, or district? Is it being measured regularly? Secondly, is the time devoted to cognitively complex instruction sufficient to give all students an opportunity to develop such thinking? What is the distribution of cognitively simple–complex learning activities? Does it reflect the Common Core? To answer these vital questions, we suggest that school leaders consider rigor as a leading indicator (Mankiw, 2007), a metric for which data are regularly collected and monitored, and which informs on the general academic health of the school.

Finally, schools of education must re-consider how they will prepare pre-service teachers in the design and implementation of knowledge work which will challenge students to think at cognitively complex levels. Is this an integral component of teacher preparation or will it require new thinking and learning on the part of faculty? Regarding school districts, how do instructional leaders equip current teachers with the capacity to manipulate cognitive complexity within their lessons? The answers to these questions are important as they relate directly to the value we place on rigorous thinking, on how we are preparing students to be college and career ready, and how we are training the next generation of adults to think critically. As a profession, we suggest there exists an important and urgent responsibility for all teachers to acquire as a core competency the ability to design knowledge work that spans the continuum of rigorous thinking.

Additional information

Funding

Funding. The authors received no direct funding for this research.

Notes on contributors

David D. Paige

David D. Paige is an associate professor of education at Bellarmine University and currently chairs the PhD program in Education and Social Change. Paige researches reading processes in adolescent readers as well as the role of higher order thinking in teaching and learning.

Grant S. Smith

Grant S. Smith is an assistant professor of educational research at Bellarmine University. Prior to his current appointment, he spent over 20 years in the private sector working in operations management, mergers and acquisitions, and the design and execution of feasibility research and process improvement studies. His research interests include accountability measurements and the relationship of educational outcomes and economic opportunity.

John M. Sizemore

Professor John M. Sizemore worked in the K-12 education field for 43 years. John currently serves as a program chair for the principal preparation program at Bellarmine University.