Measuring Subjective Movie Evaluation Criteria: Conceptual Foundation, Construction, and Validation of the SMEC Scales

ABSTRACT

Audiences’ movie evaluations have often been explored as effects of experiencing movies. However, little attention has been paid to the criteria viewers use when they evaluate a movie or its specific features. Adding to this, the present research introduces the idea of subjective movie evaluation criteria (SMEC), conceptualizes SMEC as the mental representation of important attitudes toward specific film features, and describes the scale construction for their measurement and its validation process. Findings from pilot work and 2 studies including over 1,500 participants provide first evidence that 8 dimensions—Story Verisimilitude, Story Innovation, Cinematography, Special Effects, Recommendation, Innocuousness, Light-heartedness, and Cognitive Stimulation—are largely determined by stable individual differences, substantially but differentially related to film-specific constructs and personality traits, and that the SMEC scales are reliable and valid instruments for measuring subjective movie evaluation criteria.

Acknowledgments

I thank Frieder Schmid for assisting in collecting the data, Carina Weinmann for her feedback on an earlier version of this article, as well as the anonymous reviewers and the editor, Jörg Matthes, for their valuable recommendations and thoughtful comments that helped to improve the article during the editorial process.

Notes

¹ See Neelamegham and Jain (1999), Linton and Petrovich (1988), or Wolling (2009), for instance, for similar categorizations.

² Burisch (1984) distinguishes three primary approaches to the development of self-report inventories (external, inductive, and deductive). In an inductive approach, no preconceived set of dimensions are derived from theory. Instead, empirical data analysis should reveal important dimensions and their interrelationships. Furthermore, whereas deductive scale constructionists aim at accommodating the initial constructs, inductive scale developers examine them (Tellegen & Waller, 2008, p. 261). Thus, the inductive approach can be characterized as theory-building rather than theory-testing. In personality psychology, there is a long tradition of inductive scale development, for example, several Big Five inventories based on the lexical approach or the Multidimensional Personality Questionnaire. For more details, the interested reader is referred to Tellegen and Waller (2008, p. 261–263), who discuss some advantages of an inductive approach, especially as a first step in the “inductive–hypothetico–deductive method” (Cattell, 1988).

³ The Kaiser-Meyer-Olkin coefficient was good (.85), measure of sample adequacy coefficients were all above .70, and the Bartlett test of sphericity was significant, thus indicating applicability of EFA. Mardia’s coefficient for multivariate skewness was not statistically significant, thus indicating no violation of multivariate normal distribution. The coefficient for multivariate kurtosis was statistically significant. However, the pattern matrix of the ML extraction correlated over .98 with the pattern matrix of a principal axis factor analysis, which requires no distributive assumptions, thus indicating no bias.

⁴ Throughout the text the composite reliability estimator Raykov’s ρ is reported because Cronbach’s α underestimates reliability in case of congeneric measurement models (Raykov, 1997).

⁵ More details on the psychometric properties of the scale are available from the author.

⁶ Freely estimating error covariance parameters or excluding items needs theoretical justification. On the one hand, the reasons for correlated error variances seem to lie in redundant item content. For instance, Item cs1* (“that the movie is thought-provoking”) and Item cs2 (“that the movie is intellectually challenging”) are obviously synonymous. In a similar vein, Items ci3 and ci4* as well as lh4* and lh5 seemed to be semantically related (for item wording see Table 2). Based on their psychometric properties, the items with asterisk* were preferred, while the competing ones were dropped. On the other hand, two items (fx4 and re3) were excluded because their loading patterns were ambiguous. For instance, re3 had only low loadings on RE (λ = .29), but also loaded similarly on LH (λ = .19). One reason might be that friends provide information concerning the movie’s entertainment value, whereas the remaining items of the RE dimension cover aspects of the movie itself. The Item fx4 seems to be a summary item for the FX dimension; it also statistically significantly cross-loaded on CI (λ = .36) and RE (λ = .12) and showed the lowest loading on FX (λ = .57) compared to the other FX-items. Therefore, it was excluded, too.

⁷ As in classical test theory, the reliability (Rel) represents the ratio of the true score variance to the observed variance. However, in LST theory, the true score variance can be further decomposed into a latent trait component representing stable individual differences (its proportion is reflected in the common consistency coefficient cCon), a latent state residual representing systematic situational influences (occasion specificity coefficient Spe), and—in case of method effects—a method factor component representing systematic variance of the measurement instrument or, as here, the test halves (method specificity coefficient mSpe). For further details, see Schneider, Otto et al. (2014) or Steyer and Schmitt (1990).