Measurement Invariance of Moral Foundations across Population Strata

Abstract

A representative sample (n = 2282) of Swedish adults completed the Moral Foundations Questionnaire, which measures moral intuitions concerning care, fairness, loyalty, authority, and purity. A subset (n = 607) completed a measure of intuitions about liberty. Measurement invariance was estimated across sex, age, education, income, left-right placement, religiosity, and party preference groups, based on multigroup confirmatory factor analyses of two-, three-, five-, six-, and eight-factor models, as well as bifactor models (with methods factors or a general factor). Acceptable configural, metric, and scalar invariance was obtained for most group comparisons, particularly based on the more complex models. The clearest exceptions were (1) configural non-invariance in comparisons involving participants with very low education or income, and (2) scalar non-invariance in comparisons of ideological groups based on three- and six-factor models but not the eight-factor model, which distinguished lifestyle liberty from government liberty.

Moral foundations theory (Graham et al., 2013; Haidt & Graham, 2007; Haidt & Joseph, 2004) suggests that intuitions about moral rightness rest upon at least five foundational moral concerns: care, fairness, loyalty, authority, and purity. The first two are “individualizing” insofar as they make individuals sacrifice their self-interest for the welfare of others, while the latter three are “binding” insofar as they bind individuals into collectives for which they sacrifice their self-interest (Haidt, 2008). Subsequently, a sixth foundation, involving concern with liberty, was proposed as a potential extension of the taxonomy (Graham et al., 2013; Haidt, 2012).

This taxonomy is widely used in personality and social psychological research, and it has shed new light a wide variety of phenomena, including prosocial behavior, the psychology of law, receptivity to misinformation, mental illness, environmentalism, and immoral behavior (e.g., Feinberg & Willer, 2013; Jonason et al., 2017; Kang et al., 2016; Nilsson et al., 2019, 2020; Silver, 2017). Nevertheless, the evidence for validity in the assessment of moral foundations based on the standard Moral Foundations Questionnaire (MFQ; Graham et al., 2011) is mixed.

Factor structure

Confirmatory factor analysis based on data collected with the original 30-item MFQ (with six items for each foundation) has indicated acceptable fit for the original five-factor model (e.g., Davies et al., 2014; Graham et al., 2011; Nilsson & Erlandsson, 2015; Yilmaz et al., 2016). Although the comparative fit index (CFI) frequently falls short of conventional fit criteria in these studies, this value is generally too low, and therefore not informative, when the root mean square error of approximation (RMSEA) of the independence model is low (<.158 is a benchmark suggested by Kenny, 2015), which is typically the case here.

At the same time, some studies have suggested that the taxonomy overestimates the number of factors. Iurino and Saucier (2020) found that the five-factor model failed to converge unproblematically across a diverse set of countries based on analyses of a 20-item short version of the original MFQ (with four items per foundation), although individualizing and binding items loaded on separate factors. Harper and Rhodes (2021) found three factors corresponding to individualizing, binding, and liberty intuitions in British voters, based on exploratory factor analyses of the original 30-item MFQ and nine items concerning lifestyle and government liberty (Iyer et al., 2012). Similarly, confirmatory factor analyses of the 30-item MFQ have shown small differences in fit between a hierarchical model with individualizing and binding superordinate factors and the standard five-factor model (Davies et al., 2014; Nilsson & Erlandsson, 2015).

By contrast, other studies have suggested that the foundations can be further subdivided. Zakharin and Bates (2021) split the binding intuitions into five factors (clan loyalty, country loyalty, hierarchy, sanctity, and purity) based on an analysis of misfit in the five-factor model in a sample of UK respondents and subsequently found that this model improved fit in three replication samples. With respect to liberty, exploratory factor analyses have shown that items concerning government liberty and lifestyle liberty load on separate factors in the US and Sweden (Iyer et al., 2012; Nilsson, Erlandsson et al., 2020). It should be noted, however, that these fine-grained models are data-driven and atheoretical, while the original moral foundations model was based on theoretical criteria for demarcating moral foundations (see Graham et al., 2013).

Zakharin and Bates (2021) also tested a series of bifactor models, including models with method factors and a general factor, which yielded further improvements in model fit. The reason for including method factors is that the original MFQ was designed to include both items that capture abstract theories about what is morally relevant (i.e., “relevance items”) and items that capture contextualized moral judgments (i.e., “judgment items”) in order that it would be more diverse in terms of item formats and less susceptible to response biases (Graham et al., 2009, 2011). It is therefore conceivable that variation in responses stems in part from the foundation that the item taps into and in part from whether it is a judgment item or a relevance item, and this can be adequately modeled through a bifactor model (Reise, 2012). It is, furthermore, conceivable that some of the variation in the responses is shared among items from different foundations, by virtue of shared response effects or social desirability effects or because there might exist a general moral disposition that affects all moral foundations. General factors of this sort are ubiquitous in personality trait data, although their nature is debated. Some researchers have argued that there is a substantive general factor of personality associated with adaptive functioning at the apex of the trait hierarchy (e.g., Rushton & Irwing, 2011), while others have argued that general factors are measurement artifacts that stem from socially desirable responding (Bäckström et al., 2009) or halo effects (Anusic et al., 2009) and vary across instruments (Hopwood et al., 2011).

Measurement invariance

A further challenge is that even if good model fit is obtained, this does not necessarily mean that the measured constructs are invariant (or equivalent) across groups or populations. Measurement invariance is a prerequisite for accurate mean-level comparisons across groups (Putnick & Bornstein, 2016). For instance, if purity items with religious connotations would load strongly on the purity factor in one group but weakly in another and the opposite was true for purity items without religious connotations, then the purity construct would have different meanings in the two groups (i.e., it would not be invariant). Because a high purity score would mean different things in the two groups, a mean-level comparison between the groups would not be meaningful. Furthermore, tests of associations between moral foundations and other constructs also implicitly assume that the moral foundations are invariant across subgroups in the sample (Putnick & Bornstein, 2016). If, for instance, purity would be non-invariant across sex, age, or education groups, then correlations between purity and other constructs could be difficult to interpret.

The literature is replete with mean-level comparisons of moral intuitions across political and religious groups, sexes, and countries, as well as tests of association between moral foundations and other constructs (e.g., Atari et al., 2020; Feinberg & Willer, 2013; Graham et al., 2011; Iyer et al., 2012; Koleva et al., 2012; Nilsson, Erlandsson et al., 2020; Nilsson et al., 2019). Yet evidence of measurement invariance is scarce. Doğruyol et al. (2019) obtained configural invariance (i.e., equivalence of model form) but not metric invariance (i.e., equivalence of item loadings on factors) in comparing Western and non-Western countries using the original MFQ, while Iurino and Saucier (2020) failed to obtain configural invariance across three sets of diverse countries with a 20-item short version of the original MFQ. Davis et al. (2016) obtained metric but not scalar invariance (i.e., equivalence in item intercepts) in a comparison of Black and White individuals in the US using the original MFQ. Furthermore, studies using network analysis have found substantial differences in the networks of the moral foundations between Iran and the US (Atari et al., 2020) and between liberals and conservatives in the US and New Zealand (Turner-Zwinkels et al., 2021). However, Atari et al. (2020) found metric invariance in 96-99% and scalar invariance in 55-86% of cases in comparisons of men and women in 67 countries. These studies generally used the original 30-item MFQ (except for one study in Turner-Zwinkels et al., 2021).

Overview of research

This article reports measurement invariance across sex, age, income, education, political self-placement, religious self-placement, and party preference groups. Swedish adults (n = 2282) completed the original MFQ (Graham et al., 2011), and a subset of them (n = 607) completed a measure of intuitions concerning liberty (Iyer et al., 2012). Measurement invariance was assessed in terms of confirmatory factor analyses based on two-, five-, and six-factor models, which are illustrated in Figure 1; and for the subset of participants, in terms of three-, six-, and eight-factor models, which are illustrated in Figure 2. The most complex models were based on the seven-factor model specified by Zakharin and Bates (2021), except that clan loyalty and country loyalty were represented by one loyalty factor to prevent multicollinearity. Follow-up analyses added method factors (relevance and judgment item types) and a general factor to the six-factor model (Zakharin & Bates, 2021), as illustrated in Figure 3, to further probe the robustness of the results. These analyses permitted an assessment of measurement invariance based on the best performing out of all theoretically plausible models (that could be properly fitted) in the literature, and of the robustness of results across model variations, without relying on the heavily criticized (Asparouhov & Muthén, 2014) approach in which a “partially invariant” model is fitted post hoc through stepwise model modifications.

Figure 1. Two-, five-, and six-factor models of the MFQ.

Figure 2. Three-, six-, and eight-factor models of the MFQ plus liberty.

Figure 3. Bifactor models with six moral intuitions factors and methods factors (left) or a general factor (right).

This research was conducted in Sweden, which is a social democratic welfare state and postindustrial democracy with a Protestant cultural heritage and one of the most secular, liberal, and egalitarian populations in the world (Inglehart & Welzel, 2010). Sweden has a multiparty system, in which the left (socialist, social democratic, and green parties) and the right (social liberal, liberal-conservative, and social conservative parties) differ primarily in terms of attitudes to equality, while social conservative (or traditionalist) ideology varies more within than between the left and the right (Nilsson, Montgomery et al., 2020). The familiar associations between right (vs. left) ideology and lower care and fairness and higher loyalty, authority, and purity (Graham et al., 2009; Haidt & Graham, 2007; Kivikangas et al., 2021) have been replicated in Sweden (Nilsson & Erlandsson, 2015). The effects were the strongest for the fairness and authority foundations, while concern with purity was low across the ideological spectrum and not uniquely associated with right (vs. left) self-placement adjusting for other foundations (Nilsson & Erlandsson, 2015), which may be explained in terms of the receding importance of traditional religiosity in Sweden (Inglehart & Welzel, 2010). The political value of blind patriotism is also particularly weakly endorsed by Swedes (Nilsson, Montgomery et al., 2020). Sweden is a good example of a modern liberal democracy, in which politics has increasingly become a means of self-expression, divorced from traditional religious and socio-economic structures (Caprara & Vecchione, 2017).

Materials and methods

Data, files for running the analyses, and supplemental documents are openly accessible: https://osf.io/38wsj

The research was carried out in compliance with all relevant regulations concerning research ethics. The Swedish law regarding ethics assessment of research on humans covers research that involves a physical operation, a manipulation, or the collection of biological materials or sensitive information that can be linked to an identified person. None of this applied to the current research. The participants were informed about the purposes and procedures of the research and provided their informed consent to participate.

Participants

Three samples that had been recruited from a nationwide panel of randomly selected Swedish adults were aggregated¹ (total N = 2282 after two participants with missing values had been excluded; 49.9% women, 50.1% men; M_age = 50.2, SD = 16.1). Quota sampling ensured national representativeness in terms of age, sex, and geographic region. Associations between moral foundations and other variables have been reported elsewhere (Nilsson, Montgomery et al., 2020; Nilsson et al., 2016, Nilsson, Erlandsson et al., 2020).

Measures

Moral foundations

Moral intuitions were measured with the original Swedish MFQ (Graham et al., 2011; Nilsson & Erlandsson, 2015). It measures each of the five foundations with three relevance items and three judgment items. All participants responded on Likert scales ranging from 0 (“Not at all relevant”) to 5 (“Extremely relevant”) for relevance items and 0 (“Completely disagree”) to 5 (“Completely agree”) for judgment items. Some participants (n = 607) responded to two relevance items and seven judgment items concerning liberty (Iyer et al., 2012). These items were placed among similar MFQ items with identical response scales. Reliabilities are reported in “Supplement 1 – Reliabilities.pdf” (https://osf.io/97yek).

Groups

For tests of the two-, five-, and six-factor models of the MFQ, the participants were categorized into two sexes (men: n = 1142; women: n = 1138), eight age groups (constructed through automatic binning; ns from 228 to 338), four education groups (ns from 149 to 632), seven income groups (ns from 120 to 573), seven ideological self-placement groups (from farthest to the left to farthest to the right; ns from 138 to 448), four religious groups (from vey non-religious to very religious; ns from 164 to 371), and nine party preference groups (ns from 56 to 331). For the tests of three-, six-, and eight-factor models (with intuitions about liberty), the participants were categorized into two sexes (men: n = 301; women: n = 305), four age groups (ns from 126 to 169), two education groups (low: n = 306; high: n = 301), three income groups (low: n = 152; medium: n = 239; high: n = 157), and three left-right placement groups (left: n = 141; center: n = 271; right: n = 192). Detailed descriptions of items and categorizations can be found in “Supplement 2 - Groups.pdf” (https://osf.io/pndhu/).

Statistical procedure

Confirmatory factor analysis was performed in AMOS 25.0 with the maximum likelihood method and items as indicators. The most complex models initially contained seven and nine factors, but they were reduced to six and eight factors respectively because multicollinearity emerged in some cases (ϕ > 1) when clan loyalty and country loyalty were modeled as distinct factors. Measurement invariance was evaluated in terms of the fit of a multigroup model (configural invariance) and the reduction of fit when factor loadings (metric invariance) and intercepts (scalar invariance) were constrained to be equal across groups. Follow-up analyses were performed based on the six-factor model of the MFQ to probe sources of configural misfit. In some cases, complex bifactor models could not be properly fitted. All reported results are based on models that did run correctly.

The analyses were based on traditional tests of strict measurement invariance rather than the novel approach of testing approximate measurement invariance (Asparouhov & Muthén, 2014; Luong & Flake, in press). This is because the goal of the research was to evaluate measures of moral intuitions rather than perform group-level comparisons. Methods that test approximate measurement invariance (e.g., the alignment method) permit meaningful group comparisons even when strict scalar invariance is not obtained.

Results

Fit statistics are shown in Table 1 (two-, five-, and six-factor models of the MFQ) and Table 2 (three-, six-, and eight-factor models of the MFQ and liberty). With respect to configural invariance, RMSEA was below .06 in every case, which indicates good fit, but the standardized root mean squared residual (SRMR) was frequently above .08, which indicates misfit (Hu & Bentler, 1999); the RMSEA of the independence model was consistently very low (.047-.096), which renders the absolute value of CFI uninformative (Kenny, 2015).

Table 1. Measurement invariance for two-, five-, and six-factor models of moral intuitions (the MFQ).

	Two factors					Five factors					Six factors
	χ^2	df	CFI	RMSEA [90% CI]	SRMR	χ^2	df	CFI	RMSEA [90% CI]	SRMR	χ^2	df	CFI	RMSEA [90% CI]	SRMR
Sex (n = 2280)
Configural	7356	808	.641	.060[.058, .061]	.0860	6801	790	.672	.058[.057, .059]	.0821	6507	780	.687	.057[.056, .058]	.0781
Metric	7447	836	.638	.059[.058, .060]	.0869	6888	815	.669	.057[.056, .058]	.0831	6599	804	.684	.056[.055, .058]	.0794
Scalar	7666	866	.628	.059[.057, .060]	.0870	7107	845	.658	.057[.056, .058]	.0832	6806	834	.674	.056[.055, .057]	.0796
Age (n = 2280)
Configural	12041	3778	.571	.031[.030, .032]	.1031	11493	3760	.599	.030[.029, .031]	.1014	11019	3696	.620	.030[.029, .030]	.0974
Metric	12087	3806	.570	.031[.030, .032]	.1027	11533	3785	.598	.030[.029, .031]	.1009	11095	3738	.618	.029[.029, .030]	.0974
Scalar	12230	3836	.564	.031[.030, .032]	.1026	11680	3815	.592	.030[.030, .030]	.1007	11332	3786	.608	.030[.029, .030]	.0980
Education (n = 1651)
Configural	6610	1798	.650	.040[.039, .041]	.1380	6127	1780	.684	.039[.037, .040]	.1360	5796	1752	.706	.037[.036, .039]	.1331
Metric	6684	1826	.647	.040[.039, .041]	.1340	6195	1805	.681	.038[.037, .039]	.1333	5874	1782	.702	.037[.036, .038]	.1311
Scalar	6865	1856	.635	.040[.039, .042]	.1366	6368	1835	.670	.039[.038, .040]	.1353	6067	1818	.691	.038[.037, .039]	.1334
Income (n = 2080)
Configural	10281	3283	.600	.032[.031, .033]	.1169	9772	3265	.629	.031[.030, .032]	.1170	9473	3210	.642	.031[.030, .031]	.1145
Metric	10327	3311	.599	.032[.031, .033]	.1169	9807	3290	.628	.031[.030, .032]	.1171	9522	3249	.642	.031[.030, .031]	.1155
Scalar	10376	3341	.598	.032[.031, .033]	.1168	9859	3320	.627	.031[.030, .032]	.1170	9603	3294	.640	.030[.030, .031]	.1157
Left-right self-placement (n = 1648)
Configural	9210	3283	.564	.033[.032, .034]	.1187	8735	3265	.598	.032[.031, .033]	.1160	8278	3210	.627	.031[.030, .032]	.1141
Metric	9275	3311	.561	.033[.032, .034]	.1191	8792	3290	.595	.032[.031, .033]	.1161	8346	3249	.625	.031[.030, .032]	.1146
Scalar	9416	3341	.553	.033[.032, .034]	.1190	8927	3320	.588	.032[.031, .033]	.1160	8595	3294	.610	.031[.031, .032]	.1143
Party (n = 1371)
Configural	10076	4273	.510	.032[.031, .032]	.1125	9633	4255	.546	.030[.030, .031]	.1097	9207	4182	.574	.030[.029, .031]	.1097
Metric	10095	4301	.510	.031[.031, .032]	.1126	9657	4280	.546	.030[.030, .031]	.1099	9265	4227	.574	.030[.029, .030]	.1102
Scalar	10211	4331	.503	.032[.031, .032]	.1122	9772	4310	.539	.030[.030, .031]	.1096	9480	4278	.561	.030[.029, .031]	.1094
Religiosity (n = 1046)
Configural	5652	1798	.571	.045[.044, .047]	.0905	5314	1780	.607	.044[.042, .045]	.0854	5061	1752	.637	.043[.041, .044]	.0831
Metric	5687	1826	.570	.045[.044, .046]	.0901	5351	1805	.605	.043[.042, .045]	.0849	5094	1782	.637	.042[.041, .044]	.0837
Scalar	5740	1856	.568	.045[.043, .046]	.0903	5404	1835	.603	.043[.042, .045]	.0850	5199	1818	.629	.042[.041, .044]	.0835

Note. Two-factor model: binding and individualizing intuitions. Five-factor model: care, fairness, loyalty, authority, and purity. Six-factor model: care, fairness, loyalty, authority, sanctity, and purity.

Table 2. Measurement invariance for three-, six-, and eight-factor models of moral intuitions (the MFQ and liberty).

	Three factors					Six factors					Eight factors
	χ^2	df	CFI	RMSEA [90% CI]	SRMR	χ^2	df	CFI	RMSEA [90% CI]	SRMR	χ^2	df	CFI	RMSEA [90% CI]	SRMR
Sex (n = 606)
Configural	4207	1398	.587	.058[.056, .060]	.1101	4040	1374	.608	.057[.055, .059]	.1102	3751	1348	.647	.054[.052, .056]	.1035
Metric	4307	1426	.577	.058[.056, .060]	.1123	4139	1407	.599	.057[.055, .059]	.1125	3862	1379	.635	.055[.053, .057]	.1061
Scalar	4438	1456	.562	.058[.056, .060]	.1125	4299	1446	.581	.057[.055, .059]	.1126	4013	1418	.619	.055[.053, .057]	.1061
Age (n = 606)
Configural	6765	2978	.502	.046[.045, .047]	.1131	6606	2948	.519	.045[.044, .047]	.1146	6336	2968	.553	.044[.042, .045]	.1107
Metric	6845	3030	.499	.046[.044, .047]	.1150	6681	2997	.516	.045[.044, .047]	.1160	6455	3005	.546	.044[.042, .045]	.1112
Scalar	6992	3087	.487	.046[.044, .047]	.1153	6829	3054	.504	.045[.044, .047]	.1162	6564	3050	.538	.044[.042, .045]	.1115
Education (n = 607)
Configural	4243	1398	.585	.058[.056, .060]	.0981	4081	1374	.605	.057[.055, .059]	.0970	3815	1348	.640	.055[.053, .057]	.0936
Metric	4290	1426	.583	.058[.056, .060]	.1010	4136	1399	.601	.057[.055, .059]	.1008	3885	1379	.635	.055[.053, .057]	.0979
Scalar	4367	1456	.576	.057[.056, .059]	.1013	4206	1429	.595	.057[.055, .059]	.1012	3978	1418	.627	.055[.053, .057]	.0982
Income (n = 548)
Configural	5278	2188	.539	.051[.049, .053]	.1290	5119	2161	.559	.050[.048, .052]	.1282	4888	2158	.593	.048[.046, .050]	.1241
Metric	5316	2232	.540	.050[.049, .052]	.1331	5162	2202	.559	.050[.048, .051]	.1308	4932	2192	.592	.048[.046, .049]	.1254
Scalar	5381	2280	.538	.050[.048, .052]	.1332	5225	2250	.557	.049[.048, .051]	.1310	4988	2234	.589	.048[.046, .049]	.1255
Left-right self-placement (n = 604)
Configural	5392	2188	.519	.049[.048, .051]	.1261	5240	2161	.538	.049[.047, .050]	.1239	5160	2158	.550	.048[.046, .050]	.1208
Metric	5476	2232	.513	.049[.048, .051]	.1247	5325	2202	.532	.049[.047, .050]	.1225	5252	2192	.541	.048[.047, .050]	.1204
Scalar	5737	2280	.481	.050[.049, .052]	.1276	5578	2250	.501	.050[.048, .051]	.1255	5347	2234	.533	.048[.046, .050]	.1202

Note. Three-factor model: binding, individualizing, and liberty intuitions. Six-factor model: care, fairness, loyalty, authority, purity, and liberty. Eight-factor model: care, fairness, loyalty, authority, sanctity, purity, lifestyle liberty, and government liberty.

Changes were generally smaller than .015 in RMSEA, .030 (metric invariance) and .015 (scalar invariance) in SRMR, and .010 in CFI, which indicates metric and scalar measurement invariance (Chen, 2007; Cheung & Rensvold, 2002). There were some exceptions. Changes in CFI indicated marginal scalar non-invariance for sex based on the five-factor model (ΔCFI = .011), education based on two-, five-, and six-factor models (ΔCFI from .011 to .012), as well as left-right and party groups based on the six-factor model (ΔCFI = .015/.013) (see Table 1). With items about liberty included, clear scalar non-invariance emerged for left-right groups based on three- and six-factor models (ΔCFI = .032/.031) but not the eight-factor model (ΔCFI = .008), and marginal scalar non-invariance emerged for sex based on all models (ΔCFI from .015 to .018) and age for three- and six-factor models (ΔCFI = .012) (see Table 2).

The mismatch between RMSEA and SRMR as indicators of configural fit can likely be explained by the fact that SRMR increases and RMSEA decreases with the number of degrees of freedom (Putnick & Bornstein, 2016). Simply reducing the number of age groups from eight to four (which reduces the degrees of freedom) yielded acceptable estimates (RMSEA = .041, SRMR = .0808; ΔCFI ≤ .004, ΔRMSEA ≤ .001, ΔSRMR ≤ .0025). Results from bifactor models with a general factor or method factors included, which are presented in Table 3, yielded further evidence of configural, metric, and scalar invariance across age groups and sexes.

Table 3. Measurement invariance for bifactor models.

	χ^2	df	CFI	RMSEA [90% CI]	SRMR
Sex (n = 2280): General factor
Configural	4282	724	.806	.046[.045, .048]	.0591
Metric	4430	775	.801	.046[.044, .047]	.0635
Scalar	4644	805	.790	.046[.044, .047]	.0637
Age (n = 2280): Method factors
Configural	6271	1704	.795	.034[.033, .035]	.0830
Metric	6569	1763	.786	.034[.033, .035]	.0818
Scalar	6663	1793	.783	.034[.033, .035]	.0817
Education (n = 1651): Method factors
Configural	3547	1704	.866	.026[.024, .027]	.1150
Metric	3682	1762	.860	.026[.025, .027]	.1105
Scalar	3834	1792	.851	.026[.025, .027]	.1125
Income (n = 2080): General factor
Configural	7479	3180	.755	.026[.025, .026]	.1082
Metric	7568	3248	.753	.025[.025, .026]	.1049
Scalar	7621	3278	.752	.025[.025, .026]	.1047
Left-right self-placement (n = 1648): Method factors
Configural	3376	1212	.837	.033[.032, .034]	.0716
Metric	3482	1267	.833	.033[.031, .034]	.0702
Scalar	3558	1297	.830	.033[.031, .034]	.0702
Party (n = 1371): Method factors
Configural	7574	4164	.712	.025[.024, .025]	.0939
Metric	7700	4237	.707	.024[.024, .025]	.0932
Scalar	7796	4267	.702	.025[.024, .026]	.0929

Post hoc analyses revealed that configural misfit with respect to education and income was caused mainly by participants with the lowest education and income. More acceptable estimates could be obtained for income (RMSEA = .033, SRMR = .0908; ΔCFI ≤ .002, ΔRMSEA ≤ .001, ΔSRMR ≤ .0006) and education (RMSEA = .042, SRMR = .0747; ΔCFI ≤ .016, ΔRMSEA ≤ .001, ΔSRMR ≤ .0041) if participants with very low income (n = 120) and education (n = 149) respectively were excluded. With all participants included, even bifactor models that could be successfully fitted (including method factors for education and a general factor for income) failed to yield adequate SRMR estimates (see Table 3). Adequate fit (on most fit indices) could be attained with all participants included only by collapsing the two highest and the two lowest groups, which reduced the number of income groups from seven to five (original model: RMSEA = .035, SRMR = .0880; ΔCFI ≤ .002, ΔRMSEA ≤ .001, ΔSRMR ≤ .0004; bifactor model: RMSEA = .029, SRMR = .0674; ΔCFI ≤ .001, ΔRMSEA ≤ .001, ΔSRMR ≤ .0002) and the number of educational groups from four to two (original model: RMSEA = .053, SRMR = .0754; ΔRMSEA ≤ .001, ΔSRMR ≤ .0042; for metric invariance, ΔCFI = .005, for scalar invariance, ΔCFI = .020; bifactor model: RMSEA = .035, SRMR = .0483; ΔRMSEA ≤ .001, ΔSRMR ≤ .0033; for metric invariance, ΔCFI = .008, for scalar invariance, ΔCFI = .014).

Similar post hoc analyses revealed that participants who scored the farthest to the left contributed the most to configural misfit. Excluding this group (n = 138) improved configural invariance (RMSEA = .032, SRMR = .0934; ΔRMSEA ≤ .001, ΔSRMR ≤ .0038; for metric invariance, ΔCFI = .002; for scalar invariance, ΔCFI = .017). Improved fit could be obtained with all participants included as well when the seven initial groups were collapsed into leftist, center, and rightist categories (RMSEA = .045, SRMR = .0929; ΔCFI ≤ .005, ΔRMSEA ≤ .001, ΔSRMR ≤ .0017). The only bifactor model for left-right orientation that could be properly fitted across all model steps (including method factors) indicated that adequate measurement invariance was obtained across these three groups (see Table 3). With respect to party preference, slightly lower SRMR for configural fit could be obtained, along with adequate metric and scalar invariance on most indices, either with a bifactor model that included method factors (see Table 3) or by collapsing party preferences into ideological groups (progressive left, social democratic, social liberal, liberal-conservative, social conservative; RMSEA = .033, SRMR = .0949; ΔRMSEA ≤ .001, ΔSRMR ≤ .0010; for metric invariance, ΔCFI = .001, for scalar invariance, ΔCFI = .018). A bifactor model comparing ideological groups produced further improvement in fit (RMSEA = .024, SRMR = .0757; ΔRMSEA ≤ .001, ΔSRMR ≤ .0003; for metric invariance, ΔCFI = .003, for scalar invariance, ΔCFI = .016). No bifactor model could be properly fitted for religious groups, most likely due the low number of cases.

Discussion

This research was the first to comprehensively test measurement invariance of moral foundations across socio-demographic, ideological, and religious groups within a country’s population. Analyses of data from nationally representative samples of Swedish adults were performed based on all known models of moral foundations. Acceptable measurement invariance was obtained in most cases, which implies that moral foundations can be meaningfully compared across population strata. There were however some notable exceptions.

Measurement invariance and Non-Invariance

SRMR did frequently indicate a degree of configural misfit while RMSEA indicated good fit. In some cases, this mismatch could be attributed simply to the large number of degrees of freedom. Most notably, adequate fit estimates were obtained for comparisons across age groups when the number of groups was reduced, which indicates that the high initial SRMR is not a cause for concern. In other cases, the high SRMR for configural fit proved to be more problematic. This was true particularly for comparisons across education and income groups, due to greater model misfit among participants who were the lowest on education and income. This finding suggests that caution should be exercised in making comparisons across groups that differ substantially in terms of education and income. The MFQ was originally developed and validated based on data from self-selected online samples with disproportionately high education (Graham et al., 2011). It is therefore possible that the standard models do not generalize well to other social groups.

It is also conceivable that the overrepresentation of immigrants from Africa, the Middle East, and Asia in low educated and unemployed segments of the Swedish population (Statistics Sweden, 2018, 2021) contributed to model misfit in these groups. Ethnicity, country of birth, or parents’ country of birth were unfortunately not measured in this research. This is a significant limitation. Some recent studies have suggested that the original moral foundations taxonomy may fail to account for the structure of moral intuitions outside of the Western cultural sphere (Atari et al., 2020; Iurino & Saucier, 2020). Future studies should therefore also take potential measurement non-invariance across, for instance, ethnic, migrant, or linguistic groups within countries across the world into consideration. Measurement invariance across population strata may be lower particularly in countries with greater social and economic disparities or greater demographic and linguistic diversity. Future studies may, furthermore, benefit from recent advances in the quantification of cultural distance between and within nations (Muthukrishna et al., 2020). Some past studies (e.g., Doğruyol et al., 2019) have relied on a crude dichotomy between WEIRD (Western, Educated, Industrialized, Rich, Democratic) and non-WEIRD populations (for a critique, see Apicella et al., 2020).

With respect to political orientation, the findings were mixed. SRMR indicated a degree of misfit for groups partitioned in terms of party preference and self-placement on a scale from left to right. Bifactor models with method factors (for judgment- and relevance items) were needed to obtain adequate configural, metric, and scalar invariance on all fit indices for the MFQ. There was, however, clear scalar non-invariance across left-right self-placement groups when the analyses were based on three- and six-factor models of the MFQ together with intuitions about liberty (Table 2). A likely explanation is that the liberty scale (Iyer et al., 2012) encompasses different types of liberty, which differ in importance along the political spectrum. Indeed, the scalar non-invariance disappeared when the analyses were based on the eight-factor model, which distinguished lifestyle liberty and freedom from the government. This finding shows that it important to subdivide intuitions about liberty into smaller components when comparisons across political groups are made.

The comparisons across sexes also yielded scalar non-invariance, albeit marginally. This result persisted across model variations. In cases such as this, a statistical method that does not require exact measurement invariance can be used to make comparisons across groups (Asparouhov & Muthén, 2014; Marsh et al., 2018). Indeed, Atari et al. (2020) were able to properly compare men and women in terms of moral foundations using this approach, and Brandt et al. (2021) recently used it to estimate relations between left-right self-placement and scales measuring authoritarian and egalitarian preferences (which are related to the moral foundations).

Although the purpose of the current research was to investigate measurement invariance, it may be noted that results of subsequent mean-level comparisons between political groups and sexes (see “Supplement 3: Demographic differences.pdf”: https://osf.io/uhjxt/) were similar to those reported in other studies. In a recent meta-analysis, Kivikangas et al. (2021) found that right (vs. left) self-placement was robustly negatively associated with the care and fairness foundations and positively associated the loyalty, authority, and purity foundations across countries, while social conservative (vs. liberal) self-placement was most strongly associated with loyalty, authority, and purity, particularly in European countries. Similarly, social democratic (vs. progressive) party preference on the left and social conservative (vs. liberal) party preference on the right were most robustly associated with intuitions concerning loyalty, authority, purity, and sanctity, while right (vs. left) self-placement and party preference were consistently associated with all foundations in the expected manner in the current research². Atari et al. (2020) found that women tended to score higher than men on the care, fairness, and purity foundations across countries. In the current study, women scored higher on the care and fairness foundations but marginally lower on the authority and sanctity foundations.

Dimensionality of moral foundations

The results of the tests of measurement invariance were similar between calculations based on two- and five-factor models of the MFQ and three- and six-factor models of the MFQ and liberty, consistent with findings indicating that the individualizing-binding distinction captures most of the variance in intuitions about care, fairness, loyalty, authority, and purity in Western countries (Davies et al., 2014; Harper & Rhodes, 2021; Nilsson & Erlandsson, 2015). The six- and eight-factor models (modified from Zakharin & Bates, 2021, and Iyer et al., 2012) tended to perform marginally better than the original models. This is consistent with the notion that religiosity and intuitions about purity may be distinct at least in highly secular countries such as Sweden (Nilsson & Erlandsson, 2015); religiosity was indeed a lot more strongly associated with sanctity than with purity in this study and the sanctity items were endorsed to a much lesser extent than the purity items were (“Supplement 3: Demographic differences.pdf”: https://osf.io/uhjxt/). These results cast doubt on the universal viability of unidimensional models of moral intuitions concerning purity and liberty, as assessed with the original scales (Graham et al., 2011; Iyer et al., 2012). They suggest that a theoretical framework with at least eight factors organized under at least three broader domains (individualizing intuitions subsuming care and fairness; binding intuitions subsuming loyalty, authority, purity, and sanctity; liberty intuitions subsuming lifestyle liberty and government liberty) could be an improvement over previous models for the original scales in terms of optimizing the combination of explanatory parsimony and fit.

At the same time, it is important to bear in mind that what is to count as a moral foundation in the first place is a theoretical question. Specific theoretical criteria for foundationhood have been proposed (Graham et al., 2013), and these are not necessarily satisfied by factors derived from empirical analyses. Depending on whether a factor is considered to represent a distinct moral foundation or not, model fit could be optimized either through modification of the taxonomy or through refinement of the measure. Indeed, Graham et al. (2013) envisioned a theory-method co-evolution of moral foundations theory, through which theoretical constructs would inspire new measures and data from measurement would guide theory development.

It is surprising that the MFQ has not been subjected to more psychometric refinement given its widespread adoption as an assessment tool in psychological science and the intent of its creators. However, Atari et al. (2022) did recently propose a first revised version of the MFQ (the MFQ-2). They broke the fairness foundation into separate equality and proportionality foundations on theoretical grounds while retaining the other four original foundations. They constructed a new pool of moral judgment items to measure the six posited foundations and thereafter refined and validated the instrument across a culturally diverse set of populations. The MFQ-2 generally exhibited desirable psychometric characteristics and outperformed the original MFQ (the MFQ-1) in terms of capturing variance in other relevant moral, political, and personality psychological constructs. Another notable finding was that the structure (or network) of the moral foundations failed to conform to the binding-individualizing distinction in some populations that are culturally distant from WEIRD contexts (e.g., Nigeria, Morocco, Peru, and Saudi Arabia). Hierarchical models that rely on this superordinate distinction may therefore not be viable in all cultural contexts.

The recent development of the MFQ-2 represents a major advance. Nevertheless, it is worth noting that some of the foundations have seemingly become narrower (and perhaps therefore more homogeneous). In the MFQ-2, the purity items focus mainly on religiously based sexual taboos (e.g., “I admire people who keep their virginity until marriage”), while the MFQ-1 contains items that focus on disgust as well (e.g., “People should not do things that are disgusting, even if no one is harmed”); and in the MFQ-2, all of the strongly loading loyalty items concern country loyalty (e.g., “Everyone should defend their country, if called upon”). The purity and loyalty foundations also exhibited the most measurement non-invariance across populations and the strongest associations with cultural distance (with higher scores among non-WEIRD populations: Atari et al., 2022; see also Saucier et al., 2015). The findings of Zakharin and Bates (2021) and the results of the current research illustrate the possibility of distinguishing between religiously based purity (or sanctity) and disgust-based purity and between loyalty to an impersonal collective, such as a nation, and loyalty to narrower communities, such as family and friends, with whom the individual has personal relationships (see also Voelkel & Brandt, 2019). Future studies might explore, for instance, whether more fine-grained hierarchical models with separate facet scales that are tailored to measure different types of purity, loyalty, and liberty intuitions could improve predictive power and measurement invariance of the MFQ-2 even more, particularly with highly secular and liberal cultural settings taken into consideration. Such a model could potentially remain faithful to the overarching theoretical structure posited by moral foundations theorists (Atari et al., 2022; Haidt, 2012) while further increasing the breadth and power of the model.

Concluding remarks

The results of this research suggest that the original MFQ permits meaningful generalizations across population strata, at least in WEIRD populations. Caution should be exercised particularly with respect to generalizations that encompass groups with very low education and income. Attention should be paid also to the distinctions between lifestyle and government liberty and between religious purity (or sanctity) and disgust-based moral judgments. Above all, researchers should seek to further synthesize theoretical analyses of the nature of moral intuitions and empirical findings from culturally, socio-demographically, and linguistically diverse populations. Sensitivity to contextual contingencies is essential for the development and proper application of increasingly sophisticated measures.

Open Scholarship

This article has earned the Center for Open Science badges for Open Data and Open Materials through Open Practices Disclosure. The data and materials are openly accessible at https://osf.io/38wsj/. To obtain the author's disclosure form, please contact the Editor.

Disclosure statement

The author reports that there are no competing interests to declare.

Data availability statement

The data set is accessible through the Open Science Framework project-page: https://osf.io/hrv4s/

Additional information

Funding

The data collection was supported by the Swedish Research Council (Vetenskapsrådet) under Grant 2014-01158 and the Swedish Foundation for Humanities and Social Sciences (Riksbankens Jubileumsfond) under Grant P14-0978:1.

Notes

1 The dataset initially included data from four samples. One sample was dropped after peer review because the Likert response scale for MFQ judgment items differed between this sample (1 to 7) and the others (0 to 5).

2 For Sample 1 (see “Supplement 2 – groups.pdf”: https://osf.io/pndhu/), more thorough analyses of associations between binding and individualizing sum scores and party preference have been reported elsewhere (Nilsson, Montgomery, et al., 2020)