A Validation Study of the Extended Relevance Scale using the D3mirt Package for R

ABSTRACT

This paper reports a validation study based on descriptive multidimensional item response theory (DMIRT), implemented in the R package D3mirt by using the ERS-C, an extended version of the Relevance subscale from the Moral Foundations Questionnaire including two new items for collectivism (17 items in total). Two latent models are created and investigated in two large samples (N = 1560 and N = 1380 US citizens). The internal investigation indicates a presence of two unidimensions, Compassion and Conformity, and a third within-multidimension of Collectivism in both samples. The external investigation show that Republicans are the highest on Conformity in the models, followed by Independents, and lastly Democrats. Democratic followers are the highest on compassion, followed by Independents and Republicans. No difference between groups on Collectivism were found. Lastly, the models succeeded in correctly identifying the Republican and Democratic voters but struggled to correctly classify the Independent.

KEYWORDS:

• Descriptive multidimensional item response theory
• psychometrics
• moral psychology
• social psychology

Moral Foundations Theory (MFT) has, since it was first formulated in Haidt and Joseph (2004), been recognized as a theoretical framework that can map the moral differences between the liberal and the conservative (Graham et al., 2009; Joseph et al., 2009). Studies using the Moral Foundations Questionnaire (MFQ), the measurement tool used to support the theory (Graham et al., 2011), showed that while liberals mostly preferred Care and Fairness, i.e., the Individualizing Foundations; values regarding compassion and equal rights, the conservative extended their moral palette to also include the foundations of Loyalty, Authority and Purity, i.e., the Binding Foundations; values related to group cohesion, disgust avoidance, religiosity, and conformity to a prevailing social order.

The focus of this study is, however, not theoretical nor on the MFQ but on one subscale in the MFQ, the Relevance subscale. The MFQ consists of two sub-scales, the just mentioned and the Judgement subscale, each holding 15 items. Both subscales measure the five moral foundations and the measurement model consists of a five-factor covarying factor model in turn motivating the introduction of the five labels just mentioned. That is, the five moral foundations are nothing less than the five covarying factors.

A recent study, however, showed that the two subscales have very different psychometric properties. More specifically, using non-parametric item response theory (non-parametric IRT), results indicated that the Relevance subscale was superior in psychometric performance, compared to the Judgment subscale, across samples and culture (Forsberg & Sjöberg, 2021). Perhaps, this outcome should not be surprising since the Relevance subscale consists of shorter, easier items that follow a very strict style format (examples are given in the materials section below), whereas the Judgement subscale consists of items that vary in length and semantic complexity. In other words, we can reasonably assume that the style format in the latter invites inconsistencies in responses, in turn making the subscale less likely to be psychometrically strong.

Moreover, another interesting finding was that the Loyalty Foundation seemed inconsistent with the Binding foundations and that the content of the Loyalty items seemed biased toward the Binding part of the model (Forsberg & Sjöberg, 2021). The Loyalty Foundation refers to our intuitive tendency to form cohesive coalitions and the moral violation felt when this cohesion is challenged (Haidt, 2012). According to theory, this is a distinctive feature of the conservative mind-set. However, in light of these new findings, our intuition was that this may not be correct. Moreover, it also seemed to be the case that the original construal of the foundation was one-sided. That is, it is the image of the group as an authoritarian, i.e., as a source of tribalism, obedience, pride, aggression, and punishment of traitors. In other words, what is lacking is the group as the authoritative, i.e., as a source of support, guidance, love, and care.

The current study

Consequently, we were, on the one hand, inspired by the performance of the Relevance subscale and, on the other hand, intrigued by the issues just mentioned. We, therefore, as part of a larger study on moral differences, set out to expand on the Relevance subscale by adding new items following the same style format. This became the Extended Relevance Scale, (ERS) that, over and above the items from the original Relevance subscale, involves the addition of new items with content on collectivism (two items; the expansion of ingroup/loyalty from MFT), individualism (six items), and religiosity (one item; expansion of the Purity foundation). Accordingly, to investigate the ERS-C, we will in this paper report a validation study, investigating the internal structure of the scale and testing the same against external criteria. However, because the results of the scale exapansion invite complexity, we will limit our study to reporting the results regarding the items from the original Relevance subscale and the two new collectivism items: the ERS-C, in which C stands for collectisvism.

The study will consist of two parts. Firstly, the internal properties of the ERS-C will be investigated. However, given the issues mentioned in the introduction, we will use descriptive multidimensional item response theory (DMIRT) (Reckase, 2009), explained below, and more specifically DMIRT restricted to three-dimensional analysis available in the D3mirt package for R (R Core Team, 2023). This was done because we wanted to assess the scale on item-level in a multidimensional latent environment, under the assumption that items can be unidimensional or within-multidimensional. However, because DMIRT is less explored on polytomous data, this approach comes with a need to empirically investigate the utility of the methodology and to contrast the results against more commonly used methods. In particular, because there is a statistical and conceptual association between the DMIRT discrimination indices and item factor loadings, exploratory factor analysis serves as one useful contrast. Moreover, because DMIRT is based on IRT, we also add non-parametric latent analysis as a second contrast. Briefly, non-parametric IRT relies on the most basic IRT assumptions (i.e., unidimensionality, monotonicity, and local independence) and can be used for exploring latent structures and psychometrically evaluating the same. The expectation was that we would find three latent dimensions, i.e., Compassion, Conformity, and Collectivism. Using DMIRT methodology would also show that while the former two are unidimensions, the latter will be within-multidimensional with the added collectivism items filling the conceptual gap mentioned above.

Second, to test the external performance of the measurement model identified in the first part of the study, we will focus on using respondents’ self-reported political affiliation (i.e., being affiliated with the Republican Party, the Democratic Party, or being registered as an independent). This allows us to investigate our model against relevant external criteria and to contrast our results against what is already known about the questionnaire and its external properties.

To achieve this, the first external analysis compares the mean trait-level between political ingroups on the three latent dimensions with the help of a one-way ANOVA. It was hypothesized that the Republicans and Democrats would mainly differ on their trait levels of Conformity, followed by Compassion; the Republicans being higher on Conformity and lower on Compassion and the latter being higher on Compassion and lower on Conformity. However, in contrast to theory, we hypothesized that no difference between the groups would be found regarding Collectivism.

The second external analysis investigated the identified models’ predictive utility. However, to offer something new, we used linear discriminant analysis (LDA) to predict political affiliation as the dependent categorical variable. It was expected that the models would be able to correctly label the Republican or the Democrat but the ability to correctly classify an independent voter was unknown.

Note, this paper aims to present the empirical results of our study and, in so doing, illustrate the utility of using DMIRT methodology for the analysis of items and tests. Thus, this is a psychometric paper first hand and we will, therefore, keep the theoretical discussion to a minimum. This also means that we will introduce our own labels based on results from the analysis so that the conclusions will be as close as possible to the observations reported.

Descriptive multidimensional item parameters

Within DMIRT, it is assumed that an item can be unidimensional, i.e., measure one dimension only, or within-multidimensional, i.e., measure more than one dimension, when situated in a multidimensional space (Reckase, 2009). To assess this charachteristic, the DMIRT analysis locates the singular point in the model space at which the item discrimination is at its highest, i.e., locates its global maxima. For this purpose, the multidimensional discrimination index (MDISC) has been proposed (Reckase, 2009; Reckase & McKinley, 1991):

(1) $\mathit{MDISC}=A_i=\sqrt{\sum _{\mathit{k}=1}^m{a}_{ik}^2}$(1)

Such that A_i represents the global maximum of discrimination for item i and a discrimination parameters on k dimensions (k = 1,2,3...m) in the multidimensional latent model (Reckase, 2009; Reckase & McKinley, 1991). Item discrimination and item factor loadings are mathematically associated (see de Ayala (2009), for a comprehensive discussion). Therefore, just as an item’s loading on a factor can be said to describe the item’s relation to the factor, the MDISC can be said to describe the item’s relation to the underlying latent dimension. To quantify the latter, however, the point of maximum discrimination, or maximum slope, must be found in the multidimensional latent space. This is achieved assuming that the model axes are orthogonal and taking the arccos angle (Reckase, 1985, 2009):

(2) $a_{\mathit{il}}=\mathit{co}{s}^{-1}=\left(\frac{{\mathit{a}}_{\mathit{il}}}{\sqrt{{\sum }_{k=1}^m{a}_{ik}^2}}\right)$(2)

Such that a_il is the angle on l number of axes that shows in what direction the point of maximum discrimination is oriented (Reckase, 1985, 2009). Therefore, reading the angle orientation in the model indicates along which dimensions the item discriminates. Accordingly, assuming a 2-dimensional space, an item located at 0^◦ from, for instance, the x-axis, and 90^◦ from the y-axis, only measures the latent dimension located at the x-axis and is, therefore, unidimensional. Similarly, an item located at 90^◦ from the x-axis, and 0^◦ from the y-axis, only measures the latent dimension located at the y-axis and is, therefore, also unidimensional. An item located at 45^◦ from both axis measures both dimensions equally well. Such an item is within-multidimensional.

To account for item difficulty, a single multidimensional difficulty index has been proposed (Reckase, 1985):

(3) $\mathit{MDIFF}=B_i=\frac{-d_i}{\sqrt{{\sum }_{k=1}^m{a}_{ik}^2}}$(3)

Where d is the intercept term and B_i represents the multidimensional distance from the origin and, therefore, indicates item difficulty assuming a multidimensional latent space (Reckase, 1985). The MDIFF is similar in interpretation to difficulty in the unidimensional IRT model.

It is important to note, however, that the MDIFF, just as the MDISC, only has meaning in the direction given by the point of maximum discrimination (Reckase, 2009; Reckase & McKinley, 1991). This implies that both these parameters cannot be directly compared between items, as would be the case in the unidimensional model. Thus, using the local maxima may therefore suggest a limitation to the DMIRT approach. However, to remedy this shortage, the direction discrimination index can be used (Reckase & McKinley, 1991):

(4) $\mathit{DDISC}=\sum _{\mathit{k}=1}^m{a}_{\mathit{ik}}\mathit{cos}{a}_{\mathit{ik}}$(4)

Such that DDISC gives the local discrimination score for an item when the item is constrained to follow a given direction (Reckase & McKinley, 1991). In the D3mirt package, the DDISC is implemented by the use of construct vectors that calculate the average direction of a subset of items, from all items down to one, and then adjust the discrimination index for all items in the analysis.

Method

Sample 1

The first sample consisted of N = 1560 participants (M_age= 23.65, SD = 14.68, female = 50%) US citizens, recruited through Prolific in three waves screening for political spectrum (conservative, moderate, and liberal), sex, and participation in any previous studies. The study took about 10 minutes to complete and all respondents were rewarded £1.34 for their efforts.

Regarding cultural characteristics of the sample, almost all respondents reported being born in the US (N = 1484), and mainly of Christian faith (45%) followed by Atheism (30%). Most of the participants identified as monocultural (54%) while 30% identified as multicultural. On a Likert scale from 1 to 10 regarding self-reported socioeconomic status, the sample reached an average score of M = 6.18 (SD = 1.79). A majority, 68%, had an undergraduate degree or higher and at the time of the data collection, 50% were employed or were starting a new job within the next month, while 14% were part-time employed, and 13% had a non-paid occupation (such as homemaker, retired or disabled). The remaining participants were either unemployed or did not fall into any occupational category.

Sample 2

The second sample consisted of N = 1380 participants (M_age = 41.17, SD = 13.92, female = 50%) US citizens, recruited through Prolific in three waves screening for political spectrum (conservative, moderate, and liberal), sex, and participation in any previous studies. The study took about 8 minutes to complete and all respondents were rewarded £1.20 for their efforts.

The cultural characteristics are very similar to Sample 1. For instance, almost all respondents reported being born in the US (N = 1295), and mainly of Christian faith (53%) followed by Atheism (32%). Most of the participants identified as monocultural (51%) while 31% identified as multicultural. The sample reached an average score of M = 5.12 (SD = 1.72) on the self-reported socioeconomic Likert scale. A majority, 72%, had an undergraduate degree or higher and at the time of the data collection, 57% were employed or were starting a new job within the next month, while 13% were part-time employed, and 12% had a non-paid occupation (such as homemaker, retired or disabled). The remaining participants were either unemployed or did not fall into any occupational category.

Materials

The extended relevance scale

The ERS consists of a set of 24 items and holds 5 content themes: Compassion, Conformity, Collectivism, Individualism, and Religiosity. The former two themes are the major themes in the scale while the other are subthemes. In this study, however, we limited the analysis to focus on the ERS-C (Compassion, Conformity, and Collectivism; Cite appendix: Item key in appendix). Of the total 17 items analyzed in this study, 15 items came from the Relevance subscale (see Graham, Nosek et al. (2011), for psychometric properties) and 2 new items, following the same style format as the original scale, were added.

The questionnaire consists of sentence completion tasks with the following instruction: “When you decide whether something is right or wrong, to what extent are the following considerations relevant to your thinking?” All items begin with the antecedent “Whether or not...” followed by the sentence completion statement. Examples of the latter from the original Relevance subscale are: “…someone suffered emotionally.” or “…someone’s action showed love for his or her country.” The two new items had content related to collectivism and read: ”…someone abandoned his or her team.” and ”…someone hurt his or her group.” Answers were given on a Likert scale ranging from 1 (not at all relevant) to 6 (extremely relevant).

The ERS items were used in both the samples mentioned above, collected roughly 1 year apart. In between data collection, no changes or additions to the ERS were made and results were expected to replicate.

Statistical procedure

Exploratory factor analysis

All statistical analysis in this paper was performed with R (R Core Team, 2023) in RStudio (RStudio Team, 2023) The investigation begins with an exploratory factor analysis (EFA), on the two samples separately using multidimensional IRT methods implemented in the mirt function from the mirt package (Chalmers, 2012). The graded response model was used as the item model and the estimation algorithm was set to use quasi-Monte Carlo expectation maximization. The number of factors was changed from one to six so that the potential theoretically coherent five factors were included in the set. Fit indices reported are the collapsed M2* statistic (Cai & Hansen, 2013) and the corresponding C2 statistic (for the case when the models do not have sufficient degrees of freedom; see Cai and Monro (2014)).

Factor loadings, sum of squares of rotated factor loadings, and factor correlation loadings are reported. To assess goodness-of-fit, the root mean square error of approximation (RMSEA) and the standardized root mean square residual (SRMSR) are used. To interpret the RMSEA, many suggestions have been made. A common practice is that an RMSEA< 0.06 implies adequate fit (Hu & Bentler, 1999). However, because the RMSEA is an unstandardized fit estimate that assesses model fit weighted by the number of degrees of freedom for the model, it is mostly useful for model comparisons that take model complexity into account (such that poor fit indicates issues related to the latter). In contrast, the SRMSR is a population statistic that calculates goodness-of-fit as the average standardized residual covariance that indicates the effect size of the misfit for the model (Maydeu-Olivares, 2017). Hence, if the SRMSR is poor, then the model is mis-specified in terms of the average residuals that are not being explained by the model. Adequate fit has been suggested to be the case when the SRMSR is ≤ 0.05. Accordingly, in this study, the SRMSR will serve as the main criteria such that the number of factors will be selected at the level when the EFA reports an SRMSR equal to or less than the upper bound.

In addition, the Tucker-Lewis fit index (TLI) and the comparative fit index (CFI) for null-model comparisons are reported. In brief, the TLI is a non-normed index that assesses the relative reduction in misfit per degree of freedom compared to a baseline, while the CFI is a normed index that indicates if the postulated model is an improvement in fit compared to a baseline model. Following Hu & Bentler (1999), acceptable fit is often set at ≥ 0.95 regarding both indices.

Mokken scale analysis

Accompanying the EFA, Mokken scale analysis (MSA; Mokken, 1971), implemented in the package mokken (Van der Ark, 2007, 2012), will be used to assess item fit to the monotone homogeneity model (HMH)¹ in the two samples separately. The MHM assumes that items meet the assumptions of unidimensionality, monotonicity, and local independence. In contrast to other models in IRT, the MHM makes no assumption of the specific shape of the functional form mapping the relation between the level of trait and item scores, over and above being monotonically non-decreasing. The MHM is, therefore, seen as nonrestrictive in comparison to most IRT models and is often referred to as type of non-parametric IRT.

The MSA procedure consists of, on the one hand, identification of latent scales in the data, and, on the other hand, the assessment of scalability using the coefficient H, denoted coefficient H_i for items, coefficient H_ij for item pairs, and H for scales (Mokken, 1971; Sijtsma & van der Ark, 2017). Regarding both items and scales, higher H indicates a higher level of agreement with the assumptions of the MHM, i.e., the level of extent to which the items or scales are reliable for ordering respondents on the level of trait. In general, fit varies between 0 ≤ H ≤ 1 and negative values indicate model violations. By convention, adequate fit for items and scales is the case when H ≥ 0.3. Regarding scales, the following heuristics have been suggested 0.3 ≤ H < 0.4 is a weak Mokken scale, 0.4 ≤ H < 0.5 is a moderate Mokken scale, and H ≥ 0.5 is a strong Mokken scale. An H < 0.3 is considered to indicate that the scale is unreliable for ordering respondents on the level of trait. In addition, the standard error should be considered in light of the former such that scale strength should be reduced if the SE indicate high uncertainty (Kuijpers et al., 2013).

Data exploration is performed with the automatic item selection procedure (AISP) that uses the assumptions of the MHM to extract Mokken scales (Mokken, 1971; Sijtsma & van der Ark, 2017). In brief, the AISP sorts items into scales using dummy coding variables to indicate scale-belonging for the items. For instance, for item i, a 1 in the AISP table indicates that i belongs to scale 1, a 2 indicates that item i belongs to scale 2, a 3 that i belongs to scale 3, and so on. The scale exploration is performed by order of a lower bound c that extract scales of maximal length in which (a) the covariance is non-negative for all item pairs i and j and (b) the H_i is ≥ c > 0 for all items i in the scale (Mokken, 1971). Items that fulfill these criteria are seen as contributing to the reliable ordering of respondents on level of trait under the assumption that the items measure the same latent construct.  Items whose H_i is < c are seen as unscalable. Following Hemker et al. (1995), the AISP was repeated while varying the lower bound c incrementally, in this case from 0 to 0.7.

Item and scale properties from the MSA are reported, and with the the help of the psych package (Revelle, 2023), the coefficient alpha (α), and McDonald’s (1999) Omega hierarchical (ω_h); the proportion of variance accredited to a single common factor and Omega total (ωt); the proportion of variance accredited to all common factors, are also included.

Note, the coefficients H and H_i are used in this study to assess the performance of potential latent scales, consisting of combinations of factors from the EFA, using the extracted scales from the AISP as the baseline scales. This allows for scale comparisons that can expose within-multidimensional latent structures in the data. More specifically, if the addition of items not selected by the AISP into a Mokken scale reduces the H and H_i, and more items fall below H_i < 0.3 compared to the baseline, then the items in the particular set do not seem to measure the same trait. If, on the other hand, the addition of items not selected by the AISP into a Mokken scale shows similar H scores, and all H_i ≥ 0.3, then these items seem to measure the same trait and this indicates the extra items are within-multidimensional.

Latent measurement models

For the three-dimensional DMIRT analysis, the package D3mirt (Forsberg, 2023) for dichotomous and polytomous data was used to create two latent measurement models, one for each sample. The model identification was performed using the dedicated function included in D3mirt and model fitting of the DMIRT model was done using the mirt function from the mirt package (Chalmers, 2012). Both of the former used quasi-Monte Carlo expectation maximization as the estimation algorithm and the graded response model as the item model. The model identification items for Model 1, from Sample 1, were CR1 (ABS = 0.091) and PR1 (ABS= 0.018), and for Model 2, from Sample 2, CR2 (ABS = 0.092) and AR1 (ABS = 0.028). For more on model identification see package vignette in Forsberg (2023).

Indices reported include the MDISC, MDIFF for items, and DDISC for the discrimination score constrained to follow the orientation set by the construct vectors. Item and construct vector location are reported with θ, for the ± 180^◦ clockwise rotation in the xz-plane, and φ, for the 0 ≤ φ ≤ 180°  angle away from the y-axis. The results of the item analysis are graphed in three-dimensional figures with vector arrows illustrating item response functions.

Group comparisons

For the second part of the study, the sample was partitioned according to respondents’ self-reported political affiliation, creating three categorical variables, or classes: Republican, Independent, and Democrat. The respondents’ trait scores from each class are graphed as spheres surrounded by a 95% confidence interval in the three-dimensional latent space using the plot function from the D3mirt package (Forsberg, 2023). To test for group-level differences in trait scores for the political groups in the samples, a one-way ANOVA was performed. Post-hoc tests of significant results from the ANOVA used Tukey honest significant difference (Tukey HSD).

Predicting categorical variables

An LDA analysis was conducted using the MASS package (Venables & Ripley, 2002) by constructing the two models and using the three categorical variables for the prediction of class membership. More specifically, LDA is a data reduction technique that looks for the linear combinations of variables that best model the differences between classes. The LDA does so under the assumption that observations come from a multivariate normal distribution sharing the same covariance. The analysis can be said to consist of two steps: the first step is to train the model and the second step is to use the model to predict class membership. Predictors consist of continuous independent variables and the dependent variable is a categorical or class variable. In this study, respondents’ trait scores from one DMIRT model, e.g., sample 1 and Model 1, were used as training data to predict respondents’ political affiliation in the other sample belonging to the other model, that is, Sample 2 and Model 2, and vice versa. This resulted in two LDA models: LDA 1 and LDA 2.

Factor loadings on the linear discrimination functions and proportion of trace are reported, along with the two models’ level of accuracy, Cohen’s κ coefficient, and the no information rate (NIR). The latter tests the assumption that a model can contribute significantly to the classification accuracy when removing the impact of the known distribution of the predictor variables. Cohen’s κ gives an estimation of the level of agreement of classification to the true class on the models. The following criteria have been suggested: a κ of < 0 is less than chance agreement, 0.01–0.20 is slight agreement, 0.21–0.40 is fair agreement, 0.41–0.60 is a moderate agreement, 0.61–0.80 is a substantial agreement, 0.81–0.99 is an almost perfect agreement. In addition, sensitivity, i.e., the probability of true positives, specificity, and the probability of true negatives are reported as well as the confusion matrix for both models.

Results

Part I

Exploratory factor analysis

In Table 1, we see that the limit for acceptable fit, following the criteria for SRMSR, is reached when assuming that three factors exist in the data. However, a notable drop in the RMSEA and the SRMSR can be observed moving from three to four factors in both samples, and the latter also seems to reach a plateau at four factors or more. Studying item loadings, however, indicates that four or more factors only add nuisance factor loadings (not reported) that, on the one hand, improve fit but, on the other hand, make the factor structure ambiguous. Assuming that the data hold three factors (Tables 2 and 3), however, did not indicate such issues. Thus, with the aforementioned in mind, also considering that EFA does not account for within-multidimensionality in the items that may cause a misfit to the factor projection, three factors were decided to be the most optimal case.

Table 1. EFA fit indices for the ERS-C.

Factors	M2	df	RMSEA	95% CI	SRMSR	TLI	CFI
Sample 1
1	699.107^a	51.000	0.090	[0.084, 0.096]	0.160	0.905	0.929
2	274.703^a	35.000	0.066	[0.059, 0.074]	0.069	0.949	0.974
3	195.469^a	20.000	0.075	[0.066, 0.085]	0.051	0.935	0.981
4	16.098^a	6.000	0.033	[0.014, 0.053]	0.043	0.987	0.999
5	390.712^b	61.000	0.059	[0.053, 0.065]	0.039	0.966	0.985
6	182.696^b	49.000	0.042	[0.035, 0.048]	0.042	0.983	0.994
Sample 2
1	702.828^a	51.000	0.096	[0.090, 0.103]	0.153	0.881	0.910
2	375.152^a	35.000	0.084	[0.076, 0.092]	0.068	0.909	0.953
3	101.439^a	20.000	0.054	[0.044, 0.065]	0.053	0.962	0.989
4	13.925^a	6.000	0.031	[0.009, 0.053]	0.041	0.988	0.999
5	316.650^b	61.000	0.055	[0.049, 0.061]	0.043	0.970	0.987
6	228.842^b	49.000	0.052	[0.045, 0.058]	0.038	0.974	0.991

Factors: Number of factors assumed in the EFA, M2: Model fit statistic, ^a: Model fit based on the collapsed M2* statistic, ^b: Model fit statistic based on the C2 statistic, RMSEA: Root-mean-square error of approximation, 95% CI: 95% confidence Interval, SRMSR: Standardized root mean squared residual, TLI: Tucker-Lewis index, CFI: Comparative fit index.

Table 2. EFA^a factor loadings for the ERS-C.

	Sample 1						Sample 2
Items	M	SD	F1	F2	F3	h2	M	SD	F1	F2	F3	h2
CR1	4.481	1.148	0.755	0.070	−0.021	0.566	4.530	1.113	0.766	0.097	−0.078	0.559
CR2	4.772	1.141	0.728	0.092	0.018	0.553	4.675	1.146	0.721	0.092	0.000	0.538
CR3	4.894	1.157	0.773	0.140	−0.052	0.590	4.776	1.176	0.723	−0.051	0.126	0.595
FR1	4.658	1.118	0.790	−0.115	−0.005	0.631	4.556	1.124	0.811	0.014	−0.093	0.611
FR2	4.662	1.120	0.698	−0.044	0.189	0.603	4.559	1.139	0.659	−0.124	0.242	0.578
FR3	5.147	1.048	0.750	−0.185	0.023	0.597	4.990	1.075	0.760	−0.122	−0.008	0.577
AR1	2.888	1.487	−0.053	0.705	0.105	0.591	2.952	1.486	−0.022	0.807	−0.028	0.622
AR2	3.354	1.423	0.092	0.763	0.037	0.631	3.412	1.400	0.067	0.778	0.032	0.650
AR3	2.899	1.364	−0.130	0.680	0.169	0.621	3.014	1.370	−0.125	0.688	0.140	0.601
AR4	4.275	1.269	0.404	0.440	0.004	0.369	4.204	1.237	0.463	0.358	0.087	0.441
PR1	3.481	1.644	0.028	0.787	0.018	0.637	3.673	1.568	0.075	0.613	0.134	0.514
PR2	3.465	1.571	0.120	0.549	0.141	0.440	3.575	1.528	0.170	0.399	0.181	0.341
PR3	2.721	1.859	−0.062	0.833	−0.080	0.628	2.864	1.894	−0.033	0.761	−0.035	0.547
LR1	3.699	1.383	−0.016	−0.049	0.948	0.837	3.846	1.350	−0.010	−0.030	0.934	0.833
LR2	3.431	1.356	0.008	0.209	0.653	0.631	3.490	1.353	0.036	0.147	0.723	0.695
LR3	3.278	1.324	−0.043	0.117	0.768	0.685	3.336	1.346	−0.091	0.084	0.789	0.664
LR4	3.764	1.357	0.113	−0.029	0.807	0.700	3.749	1.331	0.104	−0.009	0.811	0.721
SS Loadings			3.598	3.492	2.674				3.591	3.053	2.845

F1: Compassion, F2: Fairness, F3: Conformity, SS Loadings: The sum of squares rotated factors loadings. ^aoblimin factor rotation.

Table 3. EFA factor correlations for the ERS-C.

	Sample 1			Sample 2
	F1	F2	F3	F1	F2	F3
F1	–	–	–	–	–	–
F2	0.024	–	–	0.069	–	–
F3	0.339	0.577	–	0.367	0.611	–

F1: Compassion, F2: Fairness, F3: Conformity.

Item factor loadings and descriptives are reported in Table 2. Overall, the distribution of factor loadings does not indicate serious cross-loading tendencies, with the possible exception found in item AR4, loading on F1 and F2 in both samples.

Studying the factor loadings and the content of the strongest loading items motivated labeling factor F1 Compassion, factor F2 Conformity, and factor F3 Collectivism. It might also be mentioned that the former two factors seem to dominate the model having very similar rotated sum of squares factor loadings and are on average almost uncorrelated to each other (Table 3). The Collectivism factor, on the other hand, correlates with both dominating factors, but more so with the Conformity factor in particular.

Scale identification

Studying the scale identification table for the AISP (Tables 4 and 5), we can see a clear tendency for the items to split into two scales. This is evident already at c = 0 up to about c = 0.4. On higher levels of c, we see substructures appearing and an increasing number of items being reported as unscalable. Studying the item content, however, suggests that these substructures may be the result of strong item content similarity and not necessarily differences in latent variables.

Table 4. AISP for the ERS-C in sample 1.

								c
Items	0	0.5	0.1	0.15	0.2	0.25	0.3	0.35	0.4	0.45	0.5	0.5	0.6	0.65	0.7
CR1	1	1	1	1	2	2	2	2	2	2	2	2	2	0	0
CR2	1	1	1	1	2	2	2	2	2	2	2	0	0	0	0
CR3	2	2	2	2	2	2	2	2	2	4	4	0	0	0	0
FR1	2	2	2	2	2	2	2	2	2	2	2	2	2	0	0
FR2	2	2	2	2	2	2	2	2	2	4	4	0	0	0	0
FR3	2	2	2	2	2	2	2	2	2	4	4	0	0	0	0
AR1	1	1	1	1	1	1	1	1	1	3	3	3	3	0	0
AR2	1	1	1	1	1	1	1	1	1	3	3	3	3	0	0
AR3	1	1	1	1	1	1	1	1	1	3	3	0	0	0	0
AR4	1	1	1	1	1	1	1	0	0	0	0	0	0	0	0
PR1	1	1	1	1	1	1	1	1	1	3	3	4	0	0	0
PR2	1	1	1	1	1	1	1	1	0	0	0	0	0	0	0
PR3	1	1	1	1	1	1	1	1	1	3	3	4	0	0	0
LR1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1
LR2	1	1	1	1	1	1	1	1	1	1	1	1	0	0	0
LR3	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1
LR4	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0

c: lower bound for the AISP. Dummy variables represent scale belonging, e.g., the number 1 indicates that an item belongs to Mokken scale 1, the number 2 that the item belongs to Mokken scale 2, and so on.

Table 5. AISP for the ERS-C in sample 2.

								c
Items	0	0.5	0.1	0.15	0.2	0.25	0.3	0.35	0.4	0.45	0.5	0.5	0.6	0.65	0.7
CR1	1	1	1	1	2	2	2	2	2	3	3	3	0	0	0
CR2	1	1	1	1	2	2	2	2	2	3	0	0	0	0	0
CR3	2	2	2	2	2	2	2	2	2	4	4	0	0	0	0
FR1	2	2	2	2	2	2	2	2	2	3	3	3	0	0	0
FR2	2	2	2	2	2	2	2	2	2	4	4	0	0	0	0
FR3	2	2	2	2	2	2	2	2	2	4	0	0	0	0	0
AR1	1	1	1	1	1	1	1	1	1	2	2	2	2	0	0
AR2	1	1	1	1	1	1	1	1	1	2	2	2	2	0	0
AR3	1	1	1	1	1	1	1	1	1	2	2	0	0	0	0
AR4	1	1	1	1	1	1	1	3	0	0	0	0	0	0	0
PR1	1	1	1	1	1	1	1	1	1	2	5	0	0	0	0
PR2	1	1	1	1	1	1	1	3	0	0	5	0	0	0	0
PR3	1	1	1	1	1	1	1	1	1	2	0	0	0	0	0
LR1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1
LR2	1	1	1	1	1	1	1	1	1	1	1	1	1	0	0
LR3	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1
LR4	1	1	1	1	1	1	1	1	1	1	1	1	1	0	0

c: lower bound for the AISP. Dummy variables represent scale belonging, e.g., the number 1 indicates that an item belongs to Mokken scale 1, the number 2 that the item belongs to Mokken scale 2, and so on.

Comparing the factor structure in the EFA (Table 2) with the scale identification in the AISP (Tables 4 and 5), we find interesting similarities that should be mentioned. For instance, we see the same items separating into the two dominating structures of Compassion and Conformity in both types of analyses. In the AISP, however, Collectivism seems nested in the Conformity scale up to about c = 0.45. From that point on, however, the Collectivism substructure becomes discernible and two of the Collectivism items survive up to c = 0.7 in both samples.

Thus, the AISP replicates the same overarching latent structure in the data but also motivates a need to investigate the ambiguity in the Collectivism items further. To do this, item and test scalability was assessed on four different combinations of items (Tables 6 and 7): Scale 1 contained all items from the Compassion factor, Scale 2 contained all items from the Conformity factor, Scale 3 combined all items excluding the items from the Collectivism factor, and lastly, Scale 4 combined the items from the Compassion factor and the Collectivism factor.

Table 6. Coefficient H_i on four latent scales.

Items	Hi	SE	95% CI	Hi	SE	95% CI	Hi	SE	95% CI
CR1	0.480^a	(0.019)	[0.442, 0.517]	0.234^c	(0.016)	[0.201, 0.266]	0.351^d	(0.018)	[0.316, 0.386]
CR2	0.464^a	(0.019)	[0.428, 0.501]	0.248^c	(0.017)	[0.215, 0.280]	0.363^d	(0.018)	[0.328, 0.397]
CR3	0.459^a	(0.021)	[0.418, 0.500]	0.260^c	(0.017)	[0.226, 0.294]	0.356^d	(0.019)	[0.320, 0.393]
FR1	0.493^a	(0.018)	[0.457, 0.529]	0.175^c	(0.018)	[0.139, 0.210]	0.326^d	(0.019)	[0.289, 0.362]
FR2	0.463^a	(0.020)	[0.424, 0.503]	0.244^c	(0.017)	[0.211, 0.276]	0.386^d	(0.018)	[0.351, 0.420]
FR3	0.469^a	(0.020)	[0.431, 0.507]	0.160^c	(0.019)	[0.123, 0.197]	0.318^d	(0.019)	[0.281, 0.356]
AR1	0.433^b	(0.016)	[0.402, 0.464]	0.288^c	(0.016)	[0.258, 0.318]
AR2	0.456^b	(0.015)	[0.427, 0.485]	0.333^c	(0.013)	[0.307, 0.360]
AR3	0.450^b	(0.015)	[0.420, 0.479]	0.289^c	(0.014)	[0.261, 0.318]
AR4	0.311^b	(0.018)	[0.275, 0.347]	0.292^c	(0.014)	[0.264, 0.320]
PR1	0.455^b	(0.014)	[0.428, 0.483]	0.316^c	(0.013)	[0.290, 0.341]
PR2	0.394^b	(0.016)	[0.363, 0.425]	0.291^c	(0.014)	[0.263, 0.318]
PR3	0.400^b	(0.016)	[0.368, 0.432]	0.273^c	(0.016)	[0.242, 0.304]
LR1	0.443^b	(0.015)	[0.413, 0.473]				0.401^d	(0.015)	[0.371, 0.431]
LR2	0.449^b	(0.015)	[0.418, 0.479]				0.353^d	(0.017)	[0.320, 0.386]
LR3	0.453^b	(0.015)	[0.424, 0.482]				0.362^d	(0.017)	[0.329, 0.395]
LR4	0.411^b	(0.016)	[0.380, 0.443]				0.409^d	(0.015)	[0.378, 0.439]

^a: Scale 1, ^b: Scale 2, ^c: Scale 3, ^d: Scale 4.

Table 7. Scale properties.

Scale	M	SD	α	ωh	ωt	H	SE	95% CI
1	4.769	0.819	0.824	0.644	0.890	0.471	(0.016)	[0.439, 0.503]
2	3.387	0.981	0.879	0.670	0.909	0.424	(0.012)	[0.401, 0.447]
3	3.978	0.732	0.720	0.326	0.832	0.268	(0.011)	[0.246, 0.289]
4	4.279	0.770	0.833	0.511	0.894	0.364	(0.014)	[0.337, 0.391]

α: Coefficient alpha, ω_h: Omega hierarchical, ω_t: Omega total, H: Coefficient H, SE: Standard error for coefficient H, 95% CI: 95% confidence interval for coefficient H.

As can be seen, Scale 1 and Scale 2 perform well on all estimates (Tables 6 and 7). All consistency estimates are high and the Omega hierarchical indicates a clear presence of a common factor in both scales. The two scales are also medium in scalability, thus reliable for ordering respondents on the level of trait and all items have a scalability of ≥ 0.3.

Regarding Scale 3, however, less surprising, there is a notable drop in the Omega hierarchical (Table 7), which is somewhat observable in the alpha and the Omega total coefficient as well. Moreover, following convention, the coefficient H indicates that the scale is unreliable for ordering respondents on the level of trait and only two items in the scale achieve a scalability of ≥ 0.3 (Table 6).

In contrast, Scale 4 has alpha and Omega total scores on par with Scale 1 and Scale 2, and the common factor is approaching similar levels of strength as the just mentioned scales (Table 7). All items in the scale have an H_i ≥ 0.3 and the scale is medium in scalability, even if the H score is slightly reduced in comparison to Scale 1 and Scale 2 (Table 6). Thus, the results imply that the Collectivism items can order respondents on level of trait reliably when combined with the Conformity items or combined with the Compassion items, respectively. The difference in statistical fit is marginal.

D3mirt models

In Tables 8 and 9 and Figures 1 and 2 we can see the results of the model estimation (interactive models are available in the supplementary section). In the analysis, construct vector arrows were added following the loading structure observable in the EFA (Table 2) and item vector locations in (Figures 1 and 2). That is, items loading on Compassion, Conformity, and Collectivism were grouped to orient the respective construct vectors. Regarding the Conformity construct, however, all items loading on the factor were used except item AR4. This was because of the items’ strong within-multidimensionality tendencies, foremost observable in (Figures 1a,b and 2a,b) in which it can be seen that the item approaches a 45^◦ orientation between the x and y-axes. In other words, the item is not exclusive to Conformity, nor is it exclusive to Compassion. Note that the same item also had the highest cross-loaded tendencies of all items in the EFA analysis (Table 2). Consequently, the item is not informative to the content of the constructs in the model.

Figure 1. The ERS-C Model for sample 1 in four perspectives.

Shaded arrows represent item vectors and solid black arrows represent construct vectors.

Figure 2. The ERS-C Model for sample 2 in four perspectives.

Shaded arrows represent item vectors and solid black arrows represent construct vectors.

Table 8. DMIRT indices for sample 1.

Items	MDISC	θ	φ	MDIFF1−5	DDISC1	DDISC2	DDISC3
CR1	1.939	0.000	90.000	[−2.970, 1.156]	1.933	0.284	0.744
CR2	1.896	2.318	87.003	[−2.872, 0.685]	1.882	0.380	0.839
CR3	2.067	−2.079	86.360	[−2.732, 0.436]	2.043	0.426	0.813
FR1	2.230	0.771	103.220	[−2.693, 0.867]	2.200	−0.184	0.561
FR2	2.121	11.436	89.924	[−2.964, 0.864]	2.089	0.335	1.103
FR3	2.075	1.950	107.527	[−3.101, 0.084]	2.017	−0.323	0.448
AR1	2.028	51.864	8.130	[−1.021, 2.077]	0.046	2.023	1.386
AR2	2.220	9.086	14.392	[−1.491, 1.914]	0.397	2.208	1.509
AR3	2.174	85.291	9.403	[−1.162, 2.313]	−0.106	2.145	1.500
AR4	1.316	1.158	48.496	[−3.053, 1.553]	0.922	1.007	0.895
PR1	2.274	0.000	9.124	[−1.216, 1.413]	0.203	2.269	1.430
PR2	1.511	23.598	23.267	[−1.622, 1.871]	0.458	1.466	1.181
PR3	2.189	−89.717	4.161	[−0.181, 1.442]	−0.158	2.144	1.141
LR1	3.899	65.189	60.424	[−1.589, 1.408]	1.413	2.315	3.877
LR2	2.231	60.516	46.107	[−1.717, 1.920]	0.739	1.737	2.204
LR3	2.541	67.128	51.430	[−1.535, 2.082]	0.736	1.800	2.530
LR4	2.593	55.559	61.934	[−1.794, 1.571]	1.284	1.520	2.562

MDISC: Multidimensional discrimination index, θ: Clockwise rotation from the x-axis in the xz-plane (±180^◦), φ: the angle from the y-axis (0° ≤ φ ≤ 180°), MDIFF₁₋₅: Multidimensional difficulty on difficulty range 1 to 5, DDISC₁: Direction discrimination index for construct Compassion, DDISC₂: Direction discrimination index for construct Conformity, DDISC₃: Direction discrimination index for construct Collectivism.

Table 9. DMIRT indices for sample 2.

Items	MDISC	θ	φ	MDIFF1−5	DDISC1	DDISC2	DDISC3
CR1	1.938	−3.733	93.428	[−2.996, 1.170]	1.923	0.372	0.680
CR2	1.850	0.000	90.000	[−2.907, 0.881]	1.833	0.476	0.799
CR3	2.063	8.080	96.054	[−2.787, 0.587]	2.053	0.346	0.963
FR1	2.154	−4.757	100.415	[−2.871, 1.012]	2.134	0.150	0.572
FR2	1.985	14.102	96.110	[−2.892, 1.080]	1.944	0.343	1.053
FR3	1.990	0.466	107.003	[−3.201, 0.361]	1.960	−0.067	0.512
AR1	2.183	0.000	8.438	[−1.098, 1.963]	0.046	2.156	1.333
AR2	2.317	12.231	16.638	[−1.598, 1.873]	0.370	2.314	1.608
AR3	2.089	64.159	10.243	[−1.343, 2.121]	−0.084	2.052	1.445
AR4	1.495	9.136	59.404	[−2.997, 1.624]	1.172	1.080	1.116
PR1	1.748	26.140	22.184	[−1.657, 1.536]	0.396	1.738	1.358
PR2	1.217	26.191	36.991	[−2.119, 2.080]	0.543	1.138	1.052
PR3	1.869	−4.269	7.572	[−0.329, 1.432]	0.010	1.840	1.117
LR1	3.739	60.916	59.583	[−1.761, 1.338]	1.431	2.529	3.731
LR2	2.580	55.967	51.988	[−1.709, 1.702]	0.997	2.002	2.570
LR3	2.366	64.217	50.188	[−1.629, 2.005]	0.660	1.837	2.345
LR4	2.801	54.484	64.233	[−1.784, 1.521]	1.384	1.771	2.770

MDISC: Multidimensional discrimination index, θ: Clockwise rotation from the x-axis in the xz-plane (±180^◦), φ: the angle from the y-axis (0° ≤ φ ≤ 180°), MDIFF₁₋₅: Multidimensional difficulty on difficulty range 1 to 5, DDISC₁: Direction discrimination index for construct Compassion, DDISC₂: Direction discrimination index for construct Conformity, DDISC₃: Direction discrimination index for construct Collectivism.

Studying Tables 8 and 9 and Figures 1 and 2, we see no serious signs of misfit in the items to the assumptions of the DMIRT model. All MDISC estimates follow a similar range in both samples, also observable when comparing item vector arrow length. That is, we find no item vector arrows being obviously shorter in comparison. We also see no overly stretched MDIFF range, neither numerically nor graphically, that often accompanies low discrimination scores and high amounts of model violations. In Figures 1 and 2 it is also observable that the item locations are very similar in both samples.

Regarding the dimensionality, we find interesting patterns. Firstly, the two dominating factors from the EFA (Tables 2 and 3) are observable as clusters of item vector arrows approaching an orthogonal separation (Figures 1a,b and 2a,b). More specifically, when averaging the orientation of the items, we find Compassion at θ = 2.407^◦ and φ = 93.997^◦ in Model 1, and θ = 2.368^◦ and φ = 97.212^◦ in Model 2. It can also be seen that the Compassion items display a consistent orientation and discrimination in the model (Tables 8 and 9; Figures 1 and 2).

In the opposite direction, we find Conformity on average located at θ = 24.704^◦ and φ = 9.269^◦ in Model 1, and θ = 23.000◦ and φ = 16.241^◦ in Model 2. Looking at the Conformity items themself, however, we see more location inconsistencies in comparison to Compassion (Tables 8 and 9, Figures 1 and 2). This seems mostly to be due to some of the items having the high capacity to discriminate along the spaces spanned by both the Conformity items and the Collectivism items. This is a probable source of the less optimal fit reported in the EFA (Table 1) since the presence of within-multidimensionality in the items is likely to disrupt the variance decomposition in the factor analysis.

We should also mention item AR4, which was not included in the Conformity construct because of the items within-multidimensional tendencies. The latter can be corroborated by looking at the DDISC estimates, which are very similar regardless of construct (Tables 8 and 9). We should note, however, that neither the maximized nor constrained discrimination scores for the item are weak in comparison, therefore not implying model misfit.

Lastly, and in contrast to all of the just mentioned, we find the Collectivism items on average located at θ = 62.011^◦ and φ = 54.892^◦ in Model 1, and θ = 58.759^◦ and φ = 56.436^◦ in Model 2, occupying the area over the z-axis. However, no item clearly aligned with any axis exclusively in the model space. This is also reflected in the discrimination scores (Tables 8 and 9), i.e., all items show at least some capacity to order respondents on level of trait in all directions set by all constructs. The latter effect, however, seems stronger along the Conformity vector orientation.

Looking at the graphs we also see, in Figures 1 and 2, that the characteristic of these four items is clearly within-multidimensional with respect to both the z and y-axes (Figures 1c and 2c), and the x and z axes (Figures 1d and 2d), in both models, respectively. Even more interestingly, however, Figures 1b, 2b, Tables 8 and 9 present a semantic spectrum, relevant for our study, ordering these four items. That is, starting from the y-axis and moving toward the x-axis, we see the items falling in the order LR2 and LR3, followed by LR1 and LR4. Translating this into semantic content, we find a shift in ingroup authority construed as showing a lack of loyalty or betraying the group for the conformative, to abandoning the group and hurting the group for the compassionate.

Taken together, the results of the internal analysis in the two samples indicate a slightly complex internal structure on item-level that nevertheless seems reducible to three dimensions. In general, all items perform well but the orientation in the models shows that foremost the Collectivism items, just as with some of the items from Conformity, can order respondents on multiple traits (Tables 8 and 9). This is a likely source of misfit in the EFA (Table 1). Indeed, an EFA assuming two factors with the Collectivism items removed improved the generalizable fit in both Sample 1 (SRMSR = 0.060) and Sample 2 (SRMSR = 0.061).

Part II

ANOVA

For the next part of the study, the respondents were separated on political affiliation, resulting in three groups: respondents affiliated with the Republican Party (henceforth Republican; Sample 1: N = 420, Sample 2: N = 413), the Democratic Party (henceforth Democrat; Sample 1: N = 518, Sample 2: N = 499) or registered as an independent (henceforth Independent; Sample 1: N = 426, Sample 2: N = 399). Respondents affiliated with some other political organization, or not politically affiliated, or that had not provided any such information, were removed from this part of the analysis.

The mean values and spread for the trait levels in all groups are reported in Table 10 and the respondent’s scores are also visualized as spheres in Figures 3 to 5, oriented in the model space with the help of respondents’ trait scores used as coordinates (interactive models are available in the supplementary section). In the figures, the upper two plots graph respondent trait scores from Sample 1, and the lower two plots graph respondent trait scores from Sample 2, in which upper and lower plots are displayed rotated θ = 0^◦ and θ = 90^◦.

Figure 3. Republican respondents in the ERS-C Model.

Spheres representing trait scores for the Republicans in Sample 1 (upper) and Sample 2 (lower) plotted with a 95% confidence interval.

Table 10. Descriptives for respondent ERS-C trait scores.

	Compassion		Conformity		Collectivism
	M	SD	M	SD	M	SD
Republicans
Sample 1	−0.199	0.919	0.575	0.785	−0.073	0.889
Sample 2	−0.137	0.938	0.500	0.729	−0.013	0.810
Independents
Sample 1	−0.010	0.877	0.003	0.816	0.018	0.826
Sample 2	0.029	0.939	−0.014	0.786	0.056	0.865
Democrats
Sample 1	0.213	0.879	−0.425	0.901	0.060	0.810
Sample 2	0.123	0.865	−0.396	0.963	−0.029	0.862

Respondents sorted per sample following self-registered political affiliation.

The spheres in Figures 3 to 5 indicate a large spread within each political group. The shift in the respondent’s location on Conformity, when moving from Republicans (Figure 3) to Independents (Figure 4), and lastly to Democrats (Figure 5), is clearly discernible. The higher levels of Compassion for the Democrats (Figure 5) can also be seen when comparing plots to the Independent (Figure 4) and the Republican (Figure 3), even if the differences are not as clear as in the previous case. Studying Collectivism, however, when rotating the models 90°, shows no difference in the distribution of trait scores in the latent space.

Figure 4. Independent respondents in the ERS-C model.

Spheres representing trait scores for the Independents in Sample 1 (upper) and Sample 2 (lower) plotted with a 95% confidence interval.

Figure 5. Democratic respondents in the ERS-C Model.

Spheres representing trait scores for the Democrats in Sample 1 (upper) and Sample 2 (lower) plotted with a 95% confidence interval.

The results of the ANOVA models are reported in Table 11. Foremost, we find significant group differences in both samples on Compassion and Conformity. Post hoc tests using Tukey HSD show that Republicans are significantly lower on Compassion compared to Democrats (Sample 1: t = 7.035, p < .001, 95% CI [0.274, 0.549], Sample 2: t = 4.286, p < .001, 95% CI [0.118, 0.402]) followed by Independents (Sample 1: t = 3.091, p = .006, 95% CI [0.046, 0.333], Sample 2: t = 2.592, p = .026, 95% CI [0.016, 0.316]). Independents are, in turn, significantly lower than Democrats (t = −3.812, p < .001, 95% CI [−0.359, −0.085]) on the same trait, but only in Sample 1. Regarding Conformity, Republican affiliates are significantly higher compared to Democrats (Sample 1: t = −18.121, p < .001, 95% CI [−1.129 − 0.870], Sample 2: t = −16.001, p < .001, 95% CI [−1.027, −0.765  ]) followed by Independents (Sample 1: t = −9.904, p < .001, 95% CI [−0.708 − 0.437], Sample 2: t = −8.699, p < .001, 95% CI [−0.653, −0.375  ]). Independents are also higher compared to Democrats on the same trait in both samples (Sample 1: t = 7.779, p < .001, 95% CI [0.299, 0.556], Sample 2: t = 6.757, p < .001, 95% CI [0.249, 0.515]).

Table 11. ANOVA models on three types of moral styles.

Predictor	SS	df	F	η2	95% CI
Compassion
Sample 1	39.68	(2,1361)	25.01*	0.04	[0.02, 1.00]
Sample 2	15.40	(2,1308)	9.28*	0.01	[0.00, 1.00]
Conformity
Sample 1	231.82	(2,1361)	164.26*	0.19	[0.16, 1.00]
Sample 2	181.59	(2,1308)	128.13*	0.16	[0.13, 1.00]
Collectivism
Sample 1	4.16	(2,1361)	2.95	< 0.01	[0.00, 1.00]
Sample 2	1.76	(2,1308)	1.22	< 0.01	[0.00, 1.00]

Note: *:p _{< .}001. Regarding strength of effect: < 0.01: negligible, 0.01 ≤ η² < 0.06: small, 0.06 ≤ η² < 0.14: medium, η² ≥ 0.14: large.

Linear discriminant analysis

The overall accuracy for both LDA models is within a fair margin, with Model 2 (Accuracy = 0.540, 95% CI [0.513, 0.567], κ = 0.299) performing slightly better than Model 1 (Accuracy = 0.491, 95% CI [0.463, 0.518], κ = 0.222). The NIR was significant and very similar in both models (Model 1: NIR = 0.381, p < .001; Model 2: NIR = 0.380, p < .001). Thus, the models add to our predictive ability over and above the known distribution of classes.

Studying the Linear discriminant functions (Table 12), it can be observed that the first proportional trace accounts for almost all of the predictive strength in the models. In turn, the predictive strength seems mostly to be related to the strong negative loading from the Conformity variable on the discriminant functions LD₁ and LD₂, clearly dominating the models followed by Compassion and lastly Collectivism.

Table 12. Discriminant loadings.

	LDA 1		LDA 2
Variables	LD1	LD2	LD1	LD2
Compassion	0.477	−0.969	0.352	0.092
Conformity	−1.138	−0.325	−1.157	0.010
Collectivism	0.307	0.554	0.072	1.167
POT	0.998	0.002	0.992	0.008

LD_1,2: Linear discriminant functions. POT: Proportion of trace.

Regarding response rates, the sensitivity, specificity, and confusion matrix scores were similar in both models (Tables 13 to 15). Foremost, the capacity to discriminate between true positives or true negatives, i.e., on specificity and sensitivity, is adequate regarding the Republicans and the Democrats (Table 13). In the confusion matrices (Table 14 and 15) the fairly low rate of false positives, i.e., the rate at which the models falsely predicted a Republican or Democrat to be affiliated with some other party in the model, can be seen. For the Republicans, this was the case in 36.32% (LDA 1) and 22.62% (LDA 2) out of all possible cases in each sample, respectively. For the Democrat, this was the case in 34.87% (LDA 1) and 24.52% (LDA 2) out of all possible cases in each sample, respectively.

Table 13. Response rates for the LDA models.

	LDA 1		LDA 2
	Sensitivity	Specificity	Sensitivity	Specificity
Republicans	0.637	0.713	0.774	0.708
Independents	0.138	0.877	0.049	0.948
Democrats	0.651	0.633	0.755	0.643

LDA 1 uses Model 1 as training data to predict political affiliation in Model 2, and vice versa.

Table 14. Confusion matrix for LDA 1.

	Actual
Predicted	Republicans	Independents	Democrats
Republicans	263	142	116
Independents	54	55	58
Democrats	96	202	325

Actual: Actual class membership, Predicted: Predicted class membership.

Table 15. Confusion matrix for LDA 2.

	Actual
Predicted	Republicans	Independents	Democrats
Republicans	325	176	100
Independents	22	21	27
Democrats	73	229	391

Actual: Actual class membership, Predicted: Predicted class membership.

However, it can also be observed that the rate of false negatives for both groups was higher. More specifically, out of 521 (LDA 1) and 601 (LDA 2) predictions that a respondent was a Republican, 49.52% (LDA 1) and 45.92% (LDA 2) cases were incorrect. Likewise, out of 623 (LDA 1) and 693 (LDA 2) predictions that a respondent was a Democrat, 47.83% (LDA 1) and 43.58% (LDA 2) cases were incorrect.

Predicting Independents, however, showed very different results. The sensitivity was low, but the specificity was high in both models (Table 13). The former can be observed in Confusion matrices (Tables 14 and 15) in which the score for true positives is low and very similar to the rate of false negatives. Regarding the latter, the LDA models made fewer predictions compared to the previously mentioned. parties that a respondent was an Independent: the LDA 1 made 167 predictions and the LDA 2 model made 70 predictions. Out of these, we find 67.07% and 70.00% incorrect predictions, respectively. The rate of false positives was 86.22% (LDA 1) and 95.07% (LDA 2), respectively, out of all possible cases. Thus, we see high error rates regarding the mislabeling of Independents.

It should also be mentioned that the confusion matrices in Table 14 and 15 highlight that the reduced sensitivity seems connected to the confounding of the Independent and the Democrat, more than a confounding between the Independent and the Republican, if we study the misclassification rate following the opposite direction of the prediction path. That is, the error rate for mislabeling an Independent as a Democrat is higher than for mislabeling an Independent to be a Republican.

Discussion

This study set out to investigate the properties of the ERS-C in dual samples using DMIRT analysis, implemented in the D3mirt package (Forsberg, 2023) for R (R Core Team, 2023). Overall, the results show that DMIRT methods offer a unique insight into the structure of the item data, that the results replicate, and that the method can be used to provide a psychometric answer to empirical questions.

Looking internally, it could, for instance, be observed that neither the EFA (Table 1) nor the non-parametric IRT (Tables 4 and 5) could fully account for the complex nature of the questionnaire: the EFA reported three or more factors (Table 1) and the AISP indicated two (Tables 4 and 5). Turning to DMIRT, however, qualified the results and corroborated our expectation by showing that the data should be reduced to holding the two unidimensions of Compassion and Conformity and the within-multidimension of Collectivism (Figures 1 and 2). It could, however, be said that all methods just mentioned are complementary. We think this especially applies to the combination of MSA and DMIRT because while the former identifies latent scales the latter visualizes them, and they both utilize the assumptions of unidimensionality, monotonicity, and local independence.

Next, the psychometric analysis found support for our hypothesis that the original Loyalty foundation from the Relevance subscale could be expanded. The addition of the new collectivism items not only filled an empty space, quite literally observable in Figures 1 and 2, but it even helped form a semantic spectrum, much in line with our expectations. Namely, the content of the Collectivism items could be seen to order themself following authoritarian to authoritative-styled ingroup thinking. It should be noted that this structure replicated and would, we assume, go undiscovered using other methods.

Looking externally, we find more results that match our expectations and the literature on MFT at large. For instance, when observing group-level differences on level of trait, the strongest effects are found on Conformity followed by Compassion (Table 11). The Republicans are the highest on Conformity and the lowest on Compassion, while the opposite holds true for the Democrats and the Independents occupy a middle ground (Table 10). This resembles empirical results previously reported in Graham et al. (2011), Graham et al., (2009), and Haidt (2012), even if the analytical approach used in this paper is new. However, in contrast to the just mentioned sources, and just as we hypothesized, we find no significant difference in Collectivism between the political groups. The explanation for this outcome is, we suggest, related to the one-sided construction of the Collectivism variable that can make it seem as if the latter is more preferred by the conformative when using more common linear psychometric models.

Moreover, on the topic of group differences, we should also mention the spread of respondents’ location in the model space, visualized in (Figures 3–5). That is, even if group-level trends could be seen, both visually and statistically, the figures also illustrate that political labels may not always be informative in the individual case. In extension, this raises suspicion to the postulate that the conservative has a wider moral scope embracing all moral foundations, while the liberal has a more narrow scope foremost embracing values of compassion and fairness exclusively, as suggested by Haidt (2012). In contrast, our result indicates that political labels hide much complexity and should not be assumed to form a homogenous union.

Next, the results from the LDA show that the models can be used, at least to some extent, to correctly predict political affiliation regarding the two major American parties (Tables 13–15). This is less surprising, considering that both theory and questionnaire were originally developed for this purpose (Graham et al., 2009). However, the effect in the LDA is, just as in the ANOVA, mostly related to the Conformity variable (Table 12) in our analysis. Accordingly, the conclusion must be that what separates the Democrats and the Republicans most efficiently, from the perspective of these measurement models, is the presence of conservative values, and not values of empathy and ingroup thinking.

Regarding the independent voter, however, the models seem much more confused and the results indicate that the independent voter is barely detectable (Tables 13–15). A speculative answer to this outcome could take two routes: on the one hand, we could argue that this is the nature of the Independent, i.e., to occupy a middle moral ground and is therefore hard to discern regardless of model. On the other hand, we could also argue that the models may be insufficient for mapping the moral profile of American voters. We choose the latter.

We should also mention the observation that the independent voter seems more aligned with the moral profile of the Democrat in the LDA (Tables 14 and 15). Thus, the results indicate that the Republican tend to be more exclusive in preferring conformative values compared to the other two groups. This, however, needs more investigation and we suggest, also considering what has been said in the previous paragraph, to elaborate on the more moral content of the models.

On a last note, studying our results, it is essential to keep in mind that DMIRT assumes that within-multidimensionality is a characteristic of items and not in itself a sign of measurement errors. Thus, the within-multidimensionality reported in, for instance, item AR4 and the Collectivism items is not an indication that the items are bad. Instead, given that no other indications of model misfits can be observed (such as noticeably low discrimination scores or extreme stretch of the MDIFF range), the conclusion is that these items are good items that happen to have a within-multidimensional characteristic.

In addition, we also argue that the within-multidimensionality can even enlighten us on the psychometric characteristics of the items, not easily seen from a single analytical perspective. For example, we suggest the following. The item AR4, “…an action caused chaos or disorder”, is within-multidimensional because it is a neutral statement in terms of the dominating dimensions. That is, the content of the item can be construed from the perspective of Compassion (chaos as harm) and Conformity (chaos as a violation of order). Next, Collectivism is within-multidimensional because ingroup thinking follows with respondents’ tendency to be compassionate (collectivism as care) or confirmative (collectivism as obedience). The results, therefore, indicate that this dimension most likely functions differently because it is not fully independent, at least not when Compassion and Conformity are introduced in the model.

In extension, the notion that a trait dimension can be within-multidimensional may have implications even outside the context of this study, simply because these results hint that we should not assume ‒ nor require ‒ that all latent dimensions have similar characteristics a priori. This proposition, however, needs more investigation and Nevertheless, the general point is that it is up to the researcher to decide, in light of theory and statistical results, whether the within-multidimensionality in question is an issue or not.

We should also mention that our study was limited to an American cultural context. We, therefore, encourage cross-cultural replication following a similar design. Our method of choice for constructing our models, is also largely unexplored on data of this type. It is therefore essential to do more studies using this method, and corroborating and contrasting outcomes, not only on moral differences but also on personality differences in general.

Ethics approval

The design of this study has been reviewed by the Swedish Ethical Review Authority (Dnr 2021–01395). However, according to Swedish law, ethics approval is not required because the data collection does not imply an intervention or collection of personal data in the sense that the §§ 3–4 in the Swedish Ethics Review Act (2003:460) refers to.

Acknowledgments

We would like to express our deepest gratitude to Professor Torun Lindholm Öjmyr and Professor Mats Nilsson, Stockholm University, for their support and professional advice.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The datasets analyzed during the current study are available in the OSF repository, at https://osf.io/52ra8/?view_only=7bb2c86bad77427cad9b853412c6a029.

Supplementary material

Supplemental data for this article can be accessed online at https://doi.org/10.1080/15366367.2023.2282418.

Additional information

Funding

Research funded by Stiftelsen Elisabeth och Herman Rhodins Minne (SU FV-0045-21 and SU FV-4334-21), recipient Erik Forsberg.

Notes

1. MSA also includes the doubly monotonicity model that assumes non-intersecting response functions for the trace lines that allow for testing of invariant item ordering. The latter implies that the items in a scale follow a hierarchical order based on item difficulty. However, since invariant item ordering is not an assumption of the ERS-C, the DMM doubly monotonicity model will not be used.

Appendix

Item Key for the ERS-C

Table

Compassion
CR1	…someone suffered emotionally
CR2	…someone cared for someone weak or vulnerable
CR3	…someone was cruel
FR1	…some people were treated differently than others
FR2	…someone acted unfairly
FR3	…someone was denied his or her rights
Conformity
AR1^a	…someone’s action showed love for his or her country
AR2	…someone showed a lack of respect for authority
AR3	…someone conformed to the traditions of society
AR4	…an action caused chaos or disorder
PR1	…someone violated standards of purity and decency
PR2	…someone did something disgusting
PR3	…someone acted in a way that God would approve of
Collectivism
LR1	…someone did something to betray his or her group
LR2	…someone showed a lack of loyalty
LR3^b	…someone abandoned his or her team
LR4^b	…someone hurt his or her group

Notes: Capital letters refer to labels used in the original Relevance scale with some modifications. This was done so that items would be more easily separated when studying the results. C: compassion, F: fairness, A: authority, P: purity, and L: loyalty. ^a:This item was used in the Loyalty Foundation in the Relevance subscale. However, we have consistently found that respondents sort this item along the Conformity dimension. ^b:New item.