A Preliminary Validity of the Classroom Assessment Scoring System in Norwegian Lower-Secondary Schools

ABSTRACT

Observations seem particularly susceptible to rater error due to the level of subjectivity involved in assessment. Thus, the present paper aims to investigate: (1) inter-rater agreement (IRA) using the Classroom Assessment Scoring System – Secondary version (CLASS-S) and (2) the CLASS-S factor structure in a Norwegian context. Inter-rater agreement analysis comprise percent within one (PWO). Confirmatory factor analysis are performed to analyze the factor structure. Certified trainers scored videotaped observations. The study indicates that an acceptable IRA can be achieved in a Norwegian setting, and it reports promising results regarding the structural validity of the CLASS-S.

KEYWORDS:

• Observation
• validity
• inter-rater agreement
• multilevel analysis

Schools are pushed to implement various types of feedback and evaluation systems that aim to enhance the capacity of schools in general and classroom teachers in particular to promote student success (Hattie & Timperley, 2007; Wiliam, 2011). Systematic classroom observation instruments have been widely used to describe instructional purposes, to investigate instructional inequities, and to improve teachers’ classroom instruction based on feedback from individual, classroom or school profiles (Hamre & Pianta, 2007; Malmberg, Hagger, & Burn, 2010; Praetorius, Pauli, Reusser, Rakoczy, & Klieme, 2014). However, according to Pianta, Hamre, and Allen (2012), too many observational measurements in the past had little or no empirical evidence of their reliability or they measured components of teaching associated with student learning.

Recently, studies of the interaction between teachers and students in the classroom as a context for providing students with the opportunity to engage in insightful learning processes has gained currency in educational research. In particular, large-scale observational studies, for example, the work of Klieme and colleagues (e.g., Klieme, Pauli, & Reusser, 2009), drawing on data from the TIMMS Video, have provided insight into interactions in the classroom and their impact on students’ learning. Similarly, the Measures of Effective Teaching project in the USA, which addresses four different observational protocols including CLASS, has provided important knowledge and addressed a wide range of methodological questions related to observational data (e.g., Kane & Staiger, 2012; Kane, Kerr, & Pianta, 2014). Additionally, prominent theoretical models that describe the most important aspects of instructional quality have been developed (e.g., Kunter & Baumert, 2006; Pianta, Hamre, & Allen, 2012; Wubbels et al., 2015). Pianta and colleagues (2012) describe three subdomains of instructional quality: emotional support, classroom organization, and instructional support. This perspective is similar to others applied in previous studies (e.g., Kane & Staiger, 2012; Klieme et al., 2009; Lipowsky et al., 2009). Despite some differences, there are clear similarities between the different perspectives such that based on these domains, a hierarchical structure with underlying specific teaching dimensions and specific behavioural indicators organized within domains can be identified (Klieme et al., 2009; Pianta, Hamre, & Allen, 2012; Pianta, Hamre, & Mintz, 2012). The three key dimensions of classroom interactions have been consistently found to be associated with students’ academic and social outcomes (e.g., Allen et al., 2013; Evertson & Weinstein, 2006; Fauth, Decristan, Rieser, Klieme, & Buttner, 2014; Pianta, Hamre, & Allen, 2012).

Through the Teaching Through Interactions (TTI) framework, Hamre and Pianta (2007) developed a standardized observational measurement tool – the Classroom Assessment Scoring System (CLASS) – for observing and evaluating teacher-student interactions. There are six age-dependent versions of this instrument, ranging from the early childhood (Infant CLASS) to secondary (CLASS-S) levels. Research on CLASS has been conducted in various contexts within the USA, but few European studies and no Norwegian studies have utilized CLASS-S to assess classroom quality (Pianta, Hamre, & Mintz, 2012)

Observations seem particularly susceptible to rater error due to the level of subjectivity involved in assessment (Sandilos & DiPerna, 2011). The viability of an observational protocol and the quality of the observational data are dependent on satisfactory inter-rater-agreement (IRA) and a recognizable factor structure. Therefore, this paper has two aims. The first is to investigate whether it is possible to train observers in Norway to reliably code the interactions between teachers and students using an English protocol. The second is to investigate whether the factor structure of the CLASS-S fits a Norwegian context.

Although based on a strong international theoretical perspective, the CLASS-S was originally developed and tested in the cultural and educational context of the USA (e.g., Allen et al., 2013; Hafen et al., 2015; McCaffrey, Yuan, Savitsky, Lockwood, & Edelen, 2015). The dimensions and domains of the CLASS reflect the conceptualizations of the constructs in this educational context. Recently, the instrument has been increasingly used in other countries (e.g., Gamlem & Munthe, 2014; Malmberg et al., 2010; Virtanen et al., 2017). Its increased popularity outside the USA necessitates an investigation of the CLASS in cultural contexts or educational systems that differ from those typically found in the USA. Virtanen et al. (2017) and Malmberg et al. (2010) investigated the CLASS-S in Finish schools and among teacher students in the UK, respectively. However, no investigation of the CLASS-S has been conducted in Norway. As Pakarinen et al. (2010) discussed, there are several reasons why an investigation of the CLASS in another cultural context might add to our understanding of classroom interactions and this framework. The CLASS is expected to identify some universal characteristics of classroom interactions (Pianta, Hamre, & Mintz, 2012). Therefore, investigating the CLASS measurement in other countries and educational systems is important. Examining the dimensional structure of the CLASS in a Norwegian setting will directly test this hypothesis. Moreover, reporting validity data on the measurement in Norway will provide information on whether classroom interactions are embedded in the same correlational pattern and whether the dimensions and domains reflect the same three latent constructs as in the original context.

The Elements of the CLASS-S

Based on the TTI framework, the CLASS-S protocol classifies the quality of classroom interactions according to three overarching domains: Emotional Support, Classroom Organization, and Instructional Support. Each domain is operationalized into several age-specific dimensions of teacher-student interactions. These dimensions are grounded in previous research and theory about what promotes student learning (Allen et al., 2013; Hafen et al., 2015; Pianta, Hamre, & Allen, 2012). Each dimension is described through a set of indicators, and each indicator is connected to a list of behavioural markers (Pianta, Hamre, & Mintz, 2012). Spanning from broad, theoretically based domains to detailed behavioural markers, CLASS-S creates a multilevel theoretical framework for assessing classroom interactions that has great potential for providing nuanced data on classroom interactions. However, the measure’s complexity places great demand on rater training and IRA.

The Emotional Support domain contains dimensions that address the teacher’s ability to create and maintain a supportive social environment that fosters engagement in academic pursuits. The theoretical basis for the domain is in attachment theory and self-determination theory (Allen et al., 2013; Hafen et al., 2015; Pianta, Hamre, & Allen, 2012). The assumption is that emotionally supportive environments will make students more likely to participate in learning activities and that these activities can be enhanced by interactions that foster student autonomy, positive relations with others, and feelings of competence. In the CLASS-S protocol (Pianta, Hamre, & Mintz, 2012), Emotional Support comprises the dimensions Positive Climate, Teacher Sensitivity, and Regard for Adolescent Perspectives.

The Organization domain comprises the teacher’s skills in classroom organization. This includes the management of problem behaviour, the degree of acceptance and compliance with rules of behaviour in the classroom, and the effective use of instruction time. The theoretical foundation for this domain is in self-regulation research and studies on how teachers can help students develop skills necessary to regulate their own behaviour (Allen et al., 2013; Pianta, Hamre, & Allen, 2012). In the secondary version, the domain comprises the dimensions Behaviour Management, Productivity, and Negative Climate.

Instructional Support rates the teacher’s ability to facilitate student learning processes. Whereas Emotional Support rates the social climate in the classroom and Organization rates the teacher’s effectiveness in classroom management, Instructional Support captures the quality of the activity for student learning. This domain is based on research on children’s cognitive and language development (Hafen et al., 2015; Pianta, Hamre, & Allen, 2012). At the secondary level, Instructional Support comprises the dimensions Instructional Learning Formats, Content Understanding, Analysis and Inquiry, Quality of Feedback, and Instructional Dialogue.

In the three domains, the focus is on the classroom and teacher-level support available to students. The CLASS-S also addresses overall student functioning. Student Engagement focuses on the overall engagement level of students in the classroom (Pianta, Hamre, & Allen, 2012; Pianta, Hamre, & Mintz, 2012). Student Engagement is viewed by the CLASS developers as an outcome of the three domains (Bell et al., 2012) and is not involved in the testing of the factorial structure of the three aforementioned domains (e.g., Hafen et al., 2015).

Systematic classroom observation has the potential to generate knowledge about the quality of instruction and simultaneously provide guidance about how a given lesson can be improved (Hafen et al., 2015; Pianta, Hamre, & Allen, 2012).

Previous Research on the Reliability and Validity of the CLASS-S

One of the challenges of developing and utilizing classroom observation instruments is establishing score IRA, between raters (Olswang, Svensson, Coggins, Beilinson, & Donaldson, 2006). Given the level of subjectivity involved in assessment, observation systems are particularly susceptible to error (Sandilos & DiPerna, 2011). To assess the IRA of the scores of an observation system, two or more raters must observe the same event. If these raters produce similar results (i.e., consistent scores across raters), IRA is evident.

A few previous studies have reported IRA for the CLASS-S (e.g., Gregory, Allen, Mikami, Hafen, & Pianta, 2014; Hafen et al., 2015). Moreover, there is evidence that this instrument represents a robust understanding of classroom interactions consistent across settings (Hafen et al., 2015; Pianta, Hamre, & Allen, 2012; Sandilos & DiPerna, 2011), culture (Malmberg & Hagger, 2009), and grade level (Allen et al., 2013; Downer, Stuhlman, Schweig, Martínez, & Ruzek, 2015), although the current version of the CLASS-S (Pianta, Hamre, & Mintz, 2012) has not been investigated for consistency across cultures. However, none of the six CLASS versions have previously been tested for reliability and validity in a Scandinavian setting.

All observers undergo training. At the end of the two-day training, a reliability test is conducted. The IRA of scores is computed between the ratings of training or “master” coder and the newly trained user (Pianta, Hamre, & Mintz, 2012). The trained user has three opportunities to achieve the CLASS “certification” standards for IRA (≥ 80% agreement). Although this technique has yield high IRA estimates, the data are collected via videotaped observations in a controlled environment (Hamre, Pianta, Mashburn, & Downer, 2008; Pianta, Hamre, & Allen, 2012). The IRA reported in previous studies ranges from good to excellent (e.g., Hafen et al., 2015; McCaffrey et al., 2015; Sandilos & DiPerna, 2011) for individual dimensions and domains, indicating the complexity of IRA for complex protocols with several scoring alternatives.

Factor analyses have been performed to support both the interpretation of CLASS-S scores (e.g., Allen et al., 2013; Bell et al., 2012; Hafen et al., 2015; Sandilos & DiPerna, 2011) and the teaching theories that underlie them (Hafen et al., 2015). A common practice for generating measures of teaching is to average the scores on each dimension assigned by one or more rater across raters, segments, and lessons to obtain dimension scores, which are then aggregated to obtain a set of domain scores (e.g., Allen et al., 2013; Allen, Pianta, Mikami, & Lun, 2011; Sandilos & DiPerna, 2011). This aggregated observation score serves as a measure of the average teaching quality of each teacher across the year, and it forms the starting point for factor analysis. These procedures for interpreting scores are not specific for the different versions of the CLASS, but rather are widely used for several observation protocols (Danielson, 2013; McCaffrey et al., 2015). Recent studies have elaborated on the common practice of averaging scores and have questioned the practice of not considering the hierarchical nature of the data (Malmberg et al., 2010; McCaffrey et al., 2015). Given that the data are nested within segments nested within lessons and teachers/classrooms, it has been argued that averaging the scores across lessons and classrooms ignores variability in scores (e.g., McCaffrey et al., 2015). Ignoring this nested structure of the data can lead to mistaken conclusions about the phenomenon under study (Snijders & Bosker, 2012). Marsh et al. (2012) argued that the classroom climate (e.g., classroom interaction) is inherently a classroom-level construct and should be represented as such in statistical analyses. There are statistical reasons why it is important to use multilevel modelling when the data have a multilevel structure, for example the non-independence of responses from students within the same classroom or, as in the current study, of scores for segments within the same lesson within the same classroom. This may result in inflated Type 1 error rates (Marsh et al., 2012). Moreover, Marsh and colleagues argued that classroom climate effects should be based on group-level measures. Given that the CLASS measures classroom interactions at the classroom (group) level, it is particularly suited for multilevel analyses. The appropriate level of analysis and interpretation has been widely debated in educational research, and depends, at least in part, on the research question. However, one may argue that studies addressing classroom interaction typically evaluate whether teaching characteristics (group level) predict students’ (individual level) outcomes beyond the variance that can be explained by other individual student characteristics (Ertesvåg & Havik, 2017). In light of the recent discussion on the factor structure, it is reasonable to elaborate on it beyond the traditional approach of investigating average scores.

Applying the CLASS in the CIESL-Project

The present study, Classroom interaction for enhanced student learning (CIESL), has a longitudinal mixed-methods design (Tashakkori & Teddlie, 2010) that combines qualitative and quantitative approaches to answer the research questions. The participating lower-secondary schools and their teachers are part of a five-year programme, “Ungdomstrinn i utvikling” (UIU), initiated by the Norwegian Directorate for Education. The main aim of this programme is to make the school more practical and varied (Ministry of Education, 2011). A majority of the schools have chosen school-based professional training in classroom management. The teachers participating in UIU are provided with research-based knowledge on classroom interactions, including emotional support, instructional support, and organizational management, through seminars and workshops in school. The Norwegian Centre for Learning Environment and Behavioural Research in Education at the University of Stavanger provides support to teacher training institutions that are involved in the implementation of UIU at schools throughout the country. The main aim of the research project is to investigate how teachers implement research-based knowledge regarding classroom interaction in their teaching practices and how student outcomes are linked to teachers’ knowledge regarding classroom interaction.

In this article we focus on standardized systematic video observations to demonstrate how trained observers can reliably code the interactions between teachers and students using the CLASS-S. Moreover, we investigate the factor structure of the CLASS-S in a Norwegian context.

Thus, the questions this article aims to address are:

1. To what extent can trained observers reliably code the interactions between teachers and students?

2. How does the factor structure of the CLASS-S fit in a Norwegian context?

Methods

Sample and Procedure

Participants

The sample initially consisted of 54 teachers (aged 24–61 years, mean age = 41.89, standard deviation (SD) = 9.43) from two counties in the southern part of Norway. The teachers volunteered to be videotaped during their classroom instruction as part of a national initiative to strengthen secondary school teachers’ role in classroom processes related to student learning and development. Two teachers withdrew from the video observations; thus, the final sample included 52 teachers. Data were collected in eight schools with secondary-level students during autumn 2014 and spring 2015.

Most of the teachers were female (70%). The schools were encouraged to include teachers who varied with respect to experience, age, subject taught, and gender, if possible. Most of the teachers (60%) had been teaching for 1–7 years, 30% had been teaching for 8–17 years, and 10% had been teaching for 22–30 years. The majority of the lessons were theoretical (87%), and the remaining lessons (13%) were practical and/or elective subjects.

Students were eligible to be videotaped if they provided their parents’ and their own consent to participate in the study.

Procedure

A letter of invitation was sent to the schools describing the aim of the research and stating that participation would include video observations of the teachers who volunteered to participate.

Teachers were asked to prepare a typical class session to be facilitated in front of a camera positioned to capture the interactions between the teacher and individual students and with groups of students.

Each teacher was observed teaching the same subject in the same class in four cycles during one school year.

Each rater observed one teacher for four 45- to 60-minute cycles. For scoring purposes, each cycle was divided into two 15-minute segments (for a total of eight segments per teacher), as there is evidence that the first 30 minutes of a lesson are representative of the quality of the lesson (Joe, McClellan, & Holtzman, 2014). In total, raters observed 416 segments from 208 lessons. The CLASS-S manual (Pianta, Hamre, & Mintz, 2012) suggests scoring segments 15–20 minutes long. Accordingly, two 15-minute segments and two subsequent segments were scored, that is, 0–15 minutes and 15–30 minutes into the lesson. Procedures for scoring were developed, and instructions were distributed among raters to reduce scoring bias due to differences in the procedures. At the end of a segment, the observers assigned scores for each dimension based on the behavioural anchors provided in the CLASS-S manual before viewing a new segment. The raters did not score segments from the same lesson in series, and they did not score two segments with the same teacher consecutively. All 52 teachers were scored once by one of eight raters, and 20% of the teachers (10 teachers, 80 video clips) were scored twice by two raters to calculate the IRA. For some instances, the raters were also the data collectors (video filming), but to reduce subjectivity and rater errors, no collectors rated their own video clips.

The quality of the segments was assessed by identifying the behavioural markers for each indicator. The number, frequency, and prevalence of the behavioural markers were used to rate indicators as low, medium, or high. The indicators under each dimension were then combined in a standardized way to form a score from 1 to 7, where 1–2 indicated low quality, 3–5 medium quality, and 6–7 high quality.

To promote IRA, raters were certified according to the CLASS-S manual. The certification process consisted of a two-day training course and a test on which the raters had to score a minimum of 80% within one (PWO) over five sequences. In addition to the formalized CLASS-S certification, we used the scores of the most experienced raters as a reference to calibrate the rater corps. This was performed with four sequences before the gathered data were scored. We discovered that the complexity of the measure rendered further calibration necessary. The instructional support domain proved to be particularly challenging when seeking rater consensus.

Measures

As mentioned, the CLASS-S protocol organizes student-teacher interactions into three overarching domains – Emotional Support, Classroom Organization, and Instructional Support – and 12 dimensions. Each domain comprises three to four dimensions that observers score on a 7-point scale, with 1–2 indicating low quality, 3–5 medium quality, and 6–7 high quality. Each dimension also consists of three to four indicators (behavioural markers). The Student Engagement dimension is not incorporated into any of the domains but reflects the overall engagement level of students in the classroom (Pianta, Hamre, & Allen, 2012). Following previous CLASS-S validation studies (Allen et al., 2013; Hafen et al., 2015; Virtanen et al., 2017), the Student Engagement dimension was not included in the present study.

Rater Training

The CLASS-S was administered by eight trained and certified raters who used videos of teachers’ classroom practices. In accordance with the aforementioned CLASS procedures, the raters received a two-day training on the CLASS-S rubric and were required to score in agreement with master coders (at least 80% of the ratings were within one scale point) before starting observations. The raters also performed calibration exercises on a sample of videos of Norwegian classrooms until pairs of raters scored at least 80% of their ratings within one scale point. Project staff members who were experienced in CLASS-S scoring then reviewed the scores with the raters and discussed the disagreements between their scores.

Data Analysis

Inter-rater agreement was calculated by having two raters randomly rate 20% of the teacher’s segments (80 video clips). The primary IRA index reported in both the CLASS-S manual and in previous CLASS studies is PWO analysis (Allen et al., 2013; McCaffrey et al., 2015; Pianta, Hamre, & Allen, 2012). As PWO is the primary indicator of IRA in the CLASS-S manual, this procedure was also considered the reliability indice in the current study. As mentioned, PWO is calculated by scores that are considered to be in agreement if they fall within +/-1 point of one another. The authors of the CLASS used 0.80 as an acceptable IRA based on a PWO analysis. For two raters to achieve 80% reliability on a segment, 80% of the scores must fall within one point of one another.

Structural validity

We investigated the structural validity based on two different approaches. First, we averaged the scores as CLASS traditionally has done (e.g., Bell et al., 2012; Sandilos & DiPerna, 2011) – that is, we averaged all observers’ dimension scores for each segment and then averaged those dimension scores across the segments. Based on these data, we performed a series of alternative confirmatory factor analyses (CFAs) (Muthén & Muthén, 1998-2015). Over the past few years, this approach to the structural validation of the CLASS-S has been questioned (e.g., Hafen et al., 2015; Malmberg et al., 2010; McCaffrey et al., 2015). The traditional approach is criticized for not considering the nested structure of data with segments nested within lesson and teachers/classrooms. Therefore, we also conducted a series of two-level CFAs, correcting the standard errors for the highest level of clustering using the two-level complex commands in Mplus (Muthén & Muthén, 1998-2015). The two-level model is specified so that the items load on their substantive constructs at the within level (observation segments). At the between level (lessons/time points), the indicators of the latent constructs are the latent average of the within-level indicators based on the between-level covariance matrix (Muthén, 1991; Reise, Ventura, Nuechterlein, & Kim, 2005). To identify the model measurement, intercepts were fixed to zero at the within level and estimated at the between level (Muthén & Muthén, 1998-2015).

Similar to previous analyses of the CLASS-S factor structure (e.g., Hafen et al., 2015), our evaluation criteria included a comparative fit index (CFI) above 0.90 and close to 0.95, a root mean square error of approximation (RMSEA) under 0.10 and close to 0.060, and a standardized root mean square residual (SRMR) under 0.120 and close to 0.80. In addition, we included the Tucker Lewis Index (TLI) above 0.90 and close to 0.95 and the chi-square test (Χ²) (Hu & Bentler, 1999; Schermelleh-Engel, Moosbrugger, & Müller, 2003). Goodness of fit indices have been developed for single-level models. Therefore, we also used the guidelines described by Ryu (Ryu, 2015; Ryu & West, 2009) to test the goodness of fit (for CFI and RMSEA) of the within- and between-level models separately (see also Hox, Moerbeek, & van de Schoot, 2010). Given that data were missing for some dimensions for a few teachers, we used full information maximum likelihood to estimate our structural parameter. Maximum likelihood estimation with robust standard errors (MLR) was applied. Given that the amount of missing data was small (< 1%), it is unlikely that missing data affected the results.

Results

The first aim of the study was to measure the IRA for the CLASS-S domains and dimensions. The IRA was calculated across a sample of segments in the video observations. Agreement calculations were computed by segment, and all eight segments were calculated separately. The reliability indices for the domain and dimension scores from the PWO analysis are reported in Table 2. The mean domain scores for the eight segments within the four domains ranged from 67 to 85 PWO, with Classroom Organization and Student Engagement being highest (85 and 81 PWO, respectively). Dimension scores ranged from 30 to 100 PWO, with Instructional Learning Formats and Instructional Dialog obtaining the lowest scores (30) and Teacher Sensitivity, Behavioural Management, Negative Climate, and Quality of Feedback obtaining the highest scores (100).

Table 1. Classroom Assessment Scoring System – Secondary Domains and Dimensions.

Domains	Dimensions
Emotional support	Positive climate
	Teacher sensitivity
	Regard for adolescent persepective
Classroom organization	Behavior management
	Productivity
	Negative climate
Instructional support	Instructional learning formats
	Content understanding
	Analysis of inquiry
	Quality of feedback
	Instructional dialogue
	Student engagement

Table 2. Inter-rater agreement for Percent Within One for all eight segments.

%	1	2	3	4	5	6	7	8	M
Emotional support
Positive climate	60	40	90	70	90	60	60	50	65^a
Teacher sensitivity	80	100	80	60	80	60	80	50	74^a
Regard for adolescent perspectives	60	80	50	80	90	70	70	80	73^a
Classroom organization
Behavioral management	80	90	100	90	90	80	80	70	81^a
Productivity	90	70	80	70	80	80	70	60	75^a
Negative climate	100	80	80	100	90	100	100	100	94^a
Instructional support
Instructional learning formats	90	70	50	50	60	30	60	80	61^a
Content understanding	90	40	40	50	90	70	80	90	69^a
Analysis and inquiry	70	90	80	60	80	50	80	90	75^a
Quality of feedback	90	70	70	70	80	80	100	70	79^a
Instructional dialog	50	30	60	30	70	50	60	60	51^a

Mean for all 8 segments for the dimension

The descriptive statistics are presented in Table 3 and are based on the eight videotaped segments. The mean scores ranged from 2.38 (Analysis and Inquiry) to 6.65 (Negative Climate). All variables are considered normal with a kurtosis or skewness value +/−1.96 (Field, 2009).

Table 3. Item/scale means (M), standard deviation (SD), Skewness (Skew), Standard Error for Skewness (SE skew), Kurtosis (Kurt), Standard Error Kurtosis (SE kurt) Minimum (Min) and Maximum score (Max) for Emotional support, Classroom Organisation Instructional support, and Student Engagement.

Item	M	SD	Skew	SE skew	Kurt	SE kurt	Min	Max
Emotional support^a	4.62	0.78	−0.30	0.330	−0.70	0.650	2.79	5.88
Positive climate	4.99	0.89	−0.03	0.330	−0.75	0.650	3.25	6.88
Teacher sensitivity	5.04	0.93	−0.28	0.330	−0.54	0.650	2.88	6.50
Regard for adolescent perspective	3.83	0.76	−0.11	0.330	−1.09	0.650	2.25	5.14
Classroom Organization	6.07	0.70	−1.04	0.330	0.43	0.650	4.21	6.88
Behavioral management	6.05	0.94	−1.58	0.330	2.35	0.650	2.88	7.00
Productivity	4.52	0.89	−0.65	0.330	−0.56	0.650	3.63	6.75
Negative climate (reversed)	6.65	0.45	−0.96	0.330	−0.36	0.650	5.50	7.00
Instructional support	3.36	0.75	0.41	0.330	0.11	0.650	1.73	5.48
Instructional learning formats	4.60	0.94	−0.18	0.330	−0.52	0.650	2.50	6.63
Content understanding	3.72	0.90	−1.00	0.330	−0.41	0.650	1.63	5.50
Analysis and inquiry	2.38	0.79	0.52	0.330	0.11	0.650	1.00	4.75
Quality of feedback	3.44	0.83	0.22	0.330	−0.16	0.650	1.71	5.25
Instructional dialog	2.66	0.92	0.31	0.330	0.15	0.650	1.00	5.25

^aRating format on all dimensions and domains are 1–7. High score is positive.

Table 4 presents a matrix of the bivariate correlations between the domains of Emotional Support, Classroom Organization, and Instructional Support, along with the specific dimensional scales to measure teacher-student interaction quality.

Table 4. Correlations among domains and dimension scales.

Scale	2.	3.	4.	5.	6.	7.	8.	9.	10.	11.	12.	13.	14.
Domains
1. Emotional support	0.50**	0.72**	0.96**	0.92**	0.82**	0.49**	0.46**	0.37**	0.73**	0.59**	0.50**	0.60**	0.64**
2. Classroom organization	|	0.59**	0.51**	0.51**	0.31*	0.93**	0.90**	0.89**	0.61**	0.64**	0.30*	0.59**	0.36**
3. Instructional support		|	0.69**	0.61**	0.65**	0.46**	0.67**	0.43**	0.75**	0.92**	0.85**	0.87**	0.89**
Dimensions
4. Positive climate			|	0.87**	0.67**	0.50**	0.47**	0.41**	0.71**	0.56**	0.48**	0.57**	0.61**
5. Teacher sensitivity				|	0.58**	0.51**	0.46**	0.37**	0.69**	0.53**	0.39**	0.51**	0.47**
6. Regard for adolescent persepctive					|	0.30*	0.29*	0.21	0.56**	0.51**
7. Behavioral management						|	0.70**	0.79**	0.53**	0.48**	0.23	0.42**	0.29*
8. Productivity							|	0.73**	0.58**	0.74**	0.39**	0.71**	0.43**
9. Negative climate (reversed)								|	0.54**	0.48**	0.12	0.47**	0.20
10. Instructional learning formats									|	0.62**	0.44**	0.59**	0.53**
11. Content understanding										|	0.72**	0.87**	0.78**
12. Analysis and inquiry											|	0.67**	0.86**
13. Quaity of feedback												|	0.69**
14. Instructional dialog													|

Note. Negative Climate has been reversed.

*p < 0.05.

**p < 0.01 (type two-tailed).

One aim of this study was to examine whether a three-factor structure similar to that described for the original CLASS-S was supported by data from a Norwegian sample. First, we tested the original factor structure based on the traditional approach, which is based on measures across all segments. Confirmatory factor analyses (CFA) were conducted in which all observers’ dimension scores for each segment were averaged, and then those dimension scores were averaged across the segments. Fit indices for the original three-factor structure were provided, and a model with all dimensions loading on the same factor was tested. The CFA supported the loading of scales onto the domains, as presented earlier, with all factors loading 0.68 or higher on the relevant factor. The original three-factor model provided a relatively poor fit for all fit indices (Χ²[41] = 132.09, p < 0.001; CFI = 0.82, TLI = 076; RMSEA = 0.207, SRMR = 0.109). An inspection of the modification indices indicated that the fit was improved by allowing the error term for the dimensions Analysis and Inquiry (AI) and Instructional Dialog (ID) in the Instructional Support domain and the error terms for the dimensions Negative Climate (NC) and Behavioural Management (BM) to correlate. There is theoretical support for the assumption that AI and ID may belong to a common unobserved factor. This also applies to the dimensions NC and BM. As observed in Table 5 and Figure 1, this improved the three-factor model (Χ²[41] = 84.03, p = 0.001; CFI = 0.92, TLI = 0.88; RMSEA = 0.145, SRMR = 0.101). Moreover, high modification indices indicated a cross loading of the dimension Instructional Learning Formats (ILF) on the Emotional Support domain. Exploring a model in which ILF loaded on the Emotional Support domain, and not the Instructional Support domain, improved the fit (Χ²[40] = 72.75, p = 0.001; CFI = 0.94, TLI = 0.91; RMSEA = 0.125, SRMR = 0.095). However, as shown in Table 5, in the alternative three-factor model, freeing the aforementioned error terms improved the fit for some indices (Χ²[40] = 84.03, p < 0.001; CFI = 0.91, TLI = 0.88; RMSEA = 0.145, SRMR = 0.101), as did excluding the ILF dimension from the analysis, allowing the error terms of AI and ID and of NC and BM to correlate (Χ²[31] = 59.48, p = 0.002; CFI = 0.94, TLI = 0.91; RMSEA = 0.133, SRMR = 0.101). For all models applying the traditional approach, Productivity (P) exhibited a nonsignificant, negative, residual variance that needed to be fixed to zero in order for the models to converge.

Figure 1. Classroom Assessment Scoring System- Secondary (CLASS-S). Single level three-factor model.

Table 5. Summary of model fit indices for alternative confirmatory factor models.

	χ^2(df)	CFI	TLI	RMSEA	SRMR	SRMRb
CFA models traditional approach
Three-factor Model	84.03(41)	0.92	0.88	.145	0.101
Three-factor Model (alt1.)^a	72.75(40)	0.94	0.91	.133	0.089
Three-factor Model (alt2.)^a	67.42(39)	0.95	0.93	.118	0.086
Single-factor Model	237.50(44)	0.62	0.53	.291	0.128
Three-factor Model 11 dimensions^b	59.48(31)	0.94	0.91	.125	0.095
Single-factor Model 11 dimensions	220.18(35)	0.61	0.50	.319	0.139
Multilevel CFA models
Three-factor Model	245.70(65)	0.93	0.90	.064	0.082	0.121
Three-factor Model (alt.)^a	261.92(80)	0.90	0.87	.074	0.133	0.198
Three-factor Model 11 dimensions	212.20(79)	0.90	0.86	.082	0.148	0.250
Multilevel model fit evaluation^c
Within level		0.95		.011
Between level		0.89		.020

Note. ^aIn the alt(ernative)1 factor model Instructional Learning Formats (ILF) loads on the emotional support domain . In the alt2 factor model ILF loads both on the emotional support domain and the Instructional support domain.

^bIn the three-factor model with 11 dimensions, ILF is omitted due to cross loadings with other domains than the original.

^cModel fit evaluation are in line with Ryu (2015) and Hox et al. (2010). χ²(df) = Chi square with degree of freedom (df); CFI = Comparative Fit Index; TLI = Tucker-Lewis Index; and RMSEA = Root Mean Squared Error of Approximation; SRMR = Standardized Root Mean Square Residual (in the multilevel models the SRMR within); SRMR_b = Standardized Root Mean Square Residual for between level.

An alternative approach that considered the nesting of segments within lessons and teachers provided a better fit to the data. Increasingly, multilevel approaches to the factor structure have been applied to evaluate of the factor structure of both the secondary version (e.g., Malmberg et al., 2010; McCaffrey et al., 2015) and other versions of the CLASS (e.g., Sandilos, Shervey, DiPerna, Lei, & Cheng, 2016). As reported in Table 5 and Figure 2, the multilevel model testing the original factor structure provided acceptable fit indices on all but one of the indices (Χ²[79] = 212.20, p < 0.001; CFI = 0.93, TLI = 0.90; RMSEA = 0.064, SRMR= 0.082, SRMR_b = 0.121). This result was found given that the error terms for NC and BM and for Quality of Feedback (QF) and ILF were freed at within level and the error terms ID and AI at between level were allowed to correlate. Allowing the error terms for NC and BM to correlate resulted in a low insignificant factor loading (0.21) for NC. Although the chi-square test was significant and the SRMR value at the between level was slightly above the threshold for an acceptable fit, this model fit the data best. The alternative multilevel models with ILF loading on the Emotional Support domain and the model excluding ILF from the analysis did not provide a better fit.

Figure 2. Classroom Assessment Scoring System- Secondary (CLASS-S). Multilevel three-factor model.

Discussion

As mentioned, one of the most challenging aspects of applying classroom observational protocols such as the CLASS-S is the process of training observers in order to establish high IRA (Park, Chen, & Holtzman, 2014). Thus, the first research question to be answered in the present study was to what extent trained observers can reliably code the interactions between teachers and students using the CLASS-S in a Norwegian context.

As expected, PWO, being a broad indicator of interrater agreement, provided high and acceptable IRA. We found that the results were in accordance with those of previous studies (Bell et al., 2012; Gitomer et al., 2014; Sandilos & DiPerna, 2011; Virtanen et al., 2017).

Some of the threats to a high standard of IRA are rater experience, training processes, calibration, and the procedures used during video ratings (Bell et al., 2012; Casabianca, Lockwood, & McCaffrey, 2015; Cash, Hamre, Pianta, & Myers, 2012). The current study followed the same procedures as the CLASS-S regarding training and calibration, and the raters scored the videos according to the CLASS-S specifications. The eight raters were all employed at the university and trained and certified in the CLASS-S observational instrument, and some (6 out of 8) of the raters were experienced teachers. Calibration exercises were conducted until pairs of raters scored at least 80% of the ratings within one scale point (PWO). This is in line with previous research revealing that calibration exercises are important to identify and follow up with raters who struggle with the protocols (Casabianca et al., 2015; Cash et al., 2012). However, despite good training and calibration exercises, a previous study found that rater drift occurred over time (Casabianca et al., 2015). Casabianca and colleagues found that raters were more lenient at the beginning than at the end of observations and that the gap between the raters increased towards the end. We found no evidence of this phenomenon in our study. On the contrary, the mean domain IRA was relatively stable across the eight segments, suggesting low rater drift. A possible reason for this finding may be that the use of video observations allowed raters to score all segments and lessons from one classroom within a shorter period of time, compared with scoring live observations carried out over a whole academic year.

Some components of an observation instrument can be more difficult to rate than others. The present research revealed that the Instructional Support domain, particularly Instructional Dialogue, had the lowest IRA. This finding might imply that the Instructional Support domain is difficult to teach, learn and rate at the secondary level. It is possible that the dimensions under Instructional Support are unclear for raters. Furthermore, ratings may be affected by whether the raters have previous rating or teaching experience and what their beliefs are (Bell et al., 2012). The CLASS dimensions might reflect a way of thinking about teaching that contrasts with their prior experience as a teacher. Finally, the dimensions’ complexity could possibly result in a cognitive overload during the rating process (Joe et al., 2014). Consequently, the raters might have difficulty reaching a shared understanding of what constitutes Instructional Support in the CLASS-S protocol (Bell et al., 2012; Cash et al., 2012; Hafen et al., 2015).

The fact that some dimensions are more difficult to rate than others may have implications for how raters are selected and trained to perform observational assessments. Calibration exercises in this study involved a process of increasing awareness when discussing the discrepancy between the raters and the more experienced project observers. The IRA coefficients for the PWO were generally acceptable; they were low for all dimensions except dimensions ILF and ID.

However, the level of IRA is not determined exclusively by the process of training and calibration. Mashburn, Meyer, Allen, and Pianta (2014) found that operational procedures related to length of observation and order of presentation can impact the reliability and validity of scores. This was considered in the present study, as observation procedures were carried out according to the standard CLASS-S protocol; two 15-minute segments (15 minutes per lesson per teacher) were observed and rated non-sequentially in order to avoid carryover effects and thus reduce rater bias (Mashburn et al., 2014).

The findings from the current study indicate that the IRA is satisfactory and in line with the IRA of previous studies using the CLASS protocol. Moreover, the results support the notion that it is possible to train raters to reliably code the interactions between teachers and students in the classroom using standardized observational protocols. However, the study also illustrates, as do previous studies, the challenges of obtaining high IRA for complex protocols, such as the CLASS, comprising several dimensions that are scored from one to seven.

One may argue that in larger studies using multiple raters, the use of video observation enables the calculation of IRA across all raters, not only pairs. Of course, increasing the number of raters will most likely reduce the IRA as complexity increases (Fleiss & Cohen, 1973; Fleiss & Deese, 1971). Because CLASS-S is a relatively new, complex system with multiple levels and dimensions, it may be vulnerable to error across raters. In the current study, for calibration, all raters scored the same video segments. None of the observers were allowed to score unless they scored within 80% of the scores of the two most experienced observers using the PWO approach. This procedure was repeated for three different segments across observers until all eight observers scored within 80%. Although IRA was not tested across all raters, there is reason to believe that this calibration procedure increased the IRA tested within pairs.

The results from this study reveal that it is possible to train raters to reliably code the interactions between teachers and students using standardized observational protocols. However, some of the low IRA suggests that there remains work to be done in training raters.

Factor Structure

To answer the second research question, we investigated whether the factor structure of the CLASS-S was supported in a sample of teachers at secondary schools in Norway. First, we investigated the factor structure applying a traditional approach, averaging the dimension scores across segments and lessons and analysing them. Generally, given that some error terms were allowed to correlate, the results provided support for the current dimensions and domains described in the CLASS-S manual (Pianta, Hamre, & Mintz, 2012) and supported by research (Hafen et al., 2015). It should be noted that the Instructional Learning Formats dimension loaded higher on the Emotional Support domain than on the Instructional Support domain. Despite this, the CFA did not indicate a substantially better fit when ILF was included in the Emotional Support domain instead of the Instructional Support domain. Allowing ILF to load on both the Emotional Support domain and the Instructional Support domain improved the fit slightly. We cannot rule out the possibility that there are differences between Norwegian and US classrooms that affect the factor structure with respect to the ILF dimension. Some of the instructional behaviours typically reflected within the ILF dimensions (e.g., promoting involvement, teachers’ active display of interest and engagement in students’ work and participation, and effective engagement) may load stronger on the Emotional Support domain due to the particularly strong emphasis on positive teacher-student relationships in Norwegian schools. Given this emphasis, in comparison to US teachers, Norwegian teachers may adopt a more relational approach to instruction. Malmberg et al. (2010) omitted the ILF dimension from their factor structure to obtain good fit to the data, as did Virtanen et al. (2017). It should be noted that the sample in Malmberg and colleagues included beginning teachers. We cannot rule out the possibility that their level of experience may have affected the factor structure, as this group of teachers may have been homogenous compared to a sample of teachers with varying levels of experience. Alternatively, the findings may have been attributed to the small sample of classrooms.

Similar to Hafen et al. (2015), we found that a three-factor solution fit the data better than the alternative one-factor solution. The results for the different factor models tested were similar to or better than the corresponding fit indices Hafen et al. (2015) reported for different samples included in their study. It should be noted that Hafen and colleagues reported results only for the CFI and the RMSEA. In the current sample, the RMSEA value exceeds the cut-offs considered acceptable (Hu & Bentler, 1999). This also applied for some of the samples included by Hafen et al. (2015). Chen, Curran, Bollen, Kirby, and Paxton (2008) argued, based on a simulation study, that a cut-off of 0.05 for RMSEA is too conservative for small samples. In summary, it can be argued that the factor structure found in the current study was lower than what is commonly considered a good fit (Hu & Bentler, 1999). However, except for the RMSEA value, the values were within what is considered acceptable. Moreover, they were similar to what is found in previous studies of the CLASS-S applying the same analytical approach (Malmberg et al., 2010; McCaffrey et al., 2015).

A possible reason for the somewhat low fit in the current study, as in most previous studies, may be the underlying assumption of the traditional approach to assessing IRA and factorial structure – that is, that the independent video carries equal weight in affecting the final score (Gregory & Mikami, 2015). This assumption has been questioned (e.g., Cash & Pianta, 2014; Gregory & Mikami, 2015; Malmberg et al., 2010). An alternative assumption is that any variability displayed by a participant across time points represents measurement error (Gregory & Mikami, 2015). Another possible explanation may be rater error, that is, variability due to differences in how multiple raters score the same lesson or segment of a lesson (McCaffrey et al., 2015). In the current study, eight raters scored the 416 segments. The score of an individual rater depended, in addition to the classroom interaction he or she observed, on the general level of leniency in scoring compared with other raters. In addition, it depended on his or her specific leniencies for the observed segment, lesson, classroom, and teacher. Previous research on observation protocols indicates that this rater error is a major source of variability in scores (see, e.g., Casabianca et al., 2015; McCaffrey et al., 2015; Muthén & Muthén, 1998-2015). Accordingly, it is reasonable to assume that rater error may influence the testing of the factor structure.

However, there may also be theoretical and conceptual reasons for the somewhat low fit in the current study, as well as in previous studies. Given that teaching quality is a multidimensional concept (Pianta, Hamre, & Allen, 2012), it is unclear whether the three domains represent strictly distinct constructs. This is not only a question concerning the construct but also a question of how classroom interaction is assessed, for example, as illustrated by the cross loading of Instructional Learning Formats in the current study. From a measurement perspective, this overlap should not only manifest in high correlations between the two factors/domains of classroom interaction but also in an improvement in goodness of fit when models are used to account for items could belong to more than one factor (Scherer, Nilsen, & Jansen, 2017). An investigation of a bifactor model, which assumes a general factor and uncorrelated specific factors (Reise, 2012; Scherer et al., 2017), may shed additional light on the factor structure of the CLASS-S. Previous studies of other versions of the CLASS have indicated that a bifactor model provides a better fit than a three factor model (Hamre, Hatfield, Pianta, & Jamil, 2014). Due to the relatively small sample size of this study, testing a bifactor model was not possible. Future studies testing a bifactor model are highly welcomed.

The tested multilevel models of the proposed factor structure, taking the variability between segments and lessons into account, fit the data slightly better than the factor structure found with the traditional approach and were within what is considered acceptable. Nonetheless, some fit indices did not meet the cut-off criteria. Previous studies have reported a similar or worse fit (e.g., Malmberg et al., 2010). The low factor loading for the NC dimension at the within level when allowing the error terms of the NC and BC to correlate indicates the need for caution in the interpretation of the results. A larger sample of teachers, and thus of lessons and segments, will allow for further investigation of the factor loading of the NC. If the low factor loading is consistent, omitting the NC from the model should be considered.

Applying procedures to test model fit at each level (Hox et al., 2010; Ryu, 2015; Ryu & West, 2009) indicated that the model provided a good fit at the within level, but marginally failed to provide an acceptable fit at the between level. The discussion of factorial validity of the CLASS-S would benefit from including this procedure in future studies. One cannot assume that the factor structure relating multiple indicators to latent structure is necessarily the same at all levels (Marsh et al., 2012).

In summary, the results of the current study indicate acceptable IRA scores and an acceptable fit of the factor structure when applying both a traditional approach and a multilevel approach and considering the variability between segments and lessons. The three-factor models tested in the current study consisted of the original model (Pianta, Hamre, & Mintz, 2012) and alternative models, indicated by the modification indices. Both a traditional approach (averaging scores across segments and lessons) and a multilevel approach were tested. More studies using larger samples must be conducted before a final conclusion is drawn regarding the validity of the CLASS-S in Norway. However, the finding of the current study largely support the application of the three-factor structure suggested by Pianta, Hamre, and Mintz (2012) in Norway.

Methodological Considerations

A strength of the study is that it addresses the use of observations to investigate classroom interaction. Compared to the number of studies drawing on student and teacher reports, studies applying systematic observations are scarce, particularly in Scandinavia. Valid measurements for systematic observations of classroom interaction and the operationalization of a theoretical framework into domains and dimensions that have specific behavioural indicators have potential for providing teachers with knowledge on how to specifically translate knowledge on effective classroom interaction into practice. Moreover, teachers can receive feedback that is valuable for the further development of their teaching skills. Another strength is the testing of reliability and the original factor structure in a new cultural setting. Moreover, in addition to the traditional approach to analysing the factor structure, it included multi-level analysis that considered the hierarchical nature of the data; in contrast, most studies applying the CLASS protocol have traditionally not considered the nature of the data. In the past few years, a few studies have addressed the nested nature of the data (e.g., Malmberg et al., 2010; McCaffrey et al., 2015). Furthermore, the number lessons (208) and segments (416) was substantial, and data were collected over one academic year. However, there are some obvious limitations to the study. First, although our video-recorded observations included four lessons and eight segments for each teacher, the sample of teachers was relatively small. Multilevel models allowed for the appropriate use of the hierarchically nested data and hence prevented ecological fallacies (i.e., interpreting findings at the wrong level of the hierarchical structure [Hox et al., 2010; Snijders & Bosker, 2012]). Nevertheless, observations represent snapshots of classroom interaction and will never provide a full picture of the interaction (Muijs, 2006). Second, the sample of teachers was not drawn from a nationally representative population. It was difficult to estimate the selection bias of the teachers recruited for the study and their classrooms and to select the lessons to be included; therefore, we caution against broader generalizations of the findings before further replications of the present study are available. However, it should be noted that in a previous study using the same sample, there was no indication that the classrooms included in the observation sample contained students who were more motivated than those in the larger study not included in the observation study (Bjerga, in press).

Third, we positioned one camera with a microphone at the back of each classroom, and the teacher had a lapel microphone. Because there was no second synchronized camera positioned at the front of the classroom pointed toward the students, some of the students’ facial expressions were unavailable for coding. Fourth, our CFA did not fully confirm the a priori factor structure of the CLASS-S (Pianta, Hamre, & Allen, 2012). Our analysis provided an acceptable fit for most indices depending that some error terms were correlated. A possible reason for this may be cultural differences. Previous research on the CLASS-S in the UK (Malmberg et al., 2010) did not find support for the suggested secondary structure at the time (Pianta, La Paro, & Hamre, 2006). However, a revised version omitting some of the dimensions provided a good fit.

Final Thoughts

Observational assessment is increasingly being used to evaluate the quality of interaction between teachers and students in classroom environments. The development of standardized classroom observation systems enables the observation of large samples of teachers and classrooms. The study at hand indicates that an acceptable IRA can be achieved in a Norwegian setting, and it reports promising results regarding the structural validity of the CLASS-S. The findings provide support for the CLASS-S as a universal assessment tool across cultural contexts. The tool provides an opportunity to investigate classroom interaction across diverse contexts to inform educational theory and practise (Gregory & Mikami, 2015). Inter-rater agreement and structural validity evidence is essential but not sufficient to determine the validity of assessment scores used for a specific purpose (Sandilos et al., 2016). Future studies should continue to examine the relationship between the CLASS domain and dimensions and students’ academic and social outcomes.

In addition to investigating how the CLASS-S domains and dimensions relate to student outcomes, CLASS-S is used as a professional development tool for teachers. The current study, which supports the validity of the CLASS-S protocol, also supports the applicability of the protocol for professional development. A way to further explore the validity of the CLASS-S is to continue to examine the utility of this measure as a formative assessment that may advance or accelerate teachers’ professional development (e.g., Hamre et al., 2008).

The TTI framework offers one view into the quality of teacher-student interactions in classrooms, which are a key aspect of teachers’ jobs. The results of the current study support the validity of the CLASS-S to assess these interactions in a Norwegian context. The current study adds to the initial evidence that this is a robust model for understanding classroom interactions across cultures (Malmberg & Hagger, 2009). Moreover, the study adds to the evidence that teacher-student interactions, as observed by the CLASS-S, can reasonably be organized into domains of Emotional Support, Classroom Organization and Instructional Support across samples. This framework provides a vehicle that may help link research-based knowledge on classroom interactions to teacher evaluation and professional development.

Disclosure Statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

This work was supported by The Research Council of Norway [grant number 238003/H20].