An examination of model fit and measurement invariance of general mental ability and personality measures used in the multilingual context of the Swiss Armed Forces: A Bayesian structural equation modeling approach

ABSTRACT

Measurement invariance of psychological test batteries is an essential quality criterion when the test batteries are administered in different cultural and language contexts. The purpose of this study was to examine to what extent measurement model fit and measurement invariance across the two largest language groups in Switzerland (i.e., German and French speakers) can be assumed for selected general mental ability and personality tests used in the Swiss Armed Forces’ cadre selection process. For the model fit and invariance testing, we used Bayesian structural equation modeling (BSEM). Because the sizes of the language group samples were unbalanced, we reran the invariance testing with the subsampling procedure as a robustness check. The results showed that at least partial approximate scalar invariance can be assumed for the constructs. However, comparisons in the full sample and subsamples also showed that certain test items function differently across the language groups. The results are discussed regarding the three following issues: First, we critically discuss the applied criterion and alternative effect size measures for assessing the practical importance of non-invariances. Second, we highlight potential remedies and further testing options, that can be applied, once certain items have been detected to function differently. Third, we discuss alternative modeling and invariance testing approaches to BSEM and outline future research avenues.

KEYWORDS:

• Measurement model fit
• measurement invariance
• Bayesian structural equation modeling
• subsampling
• construct validity

What is the public significance of this article?—If psychological test batteries are meant to be valid in a cross-cultural context, two central prerequisites need to be given for the utilized measures. First, the scale items have to adequately reflect the underlying intended constructs. Second, the employed measures need to have the same psychometric properties in each cultural group. By examining these two important issues in the case of the multilingual psychological cadre assessment of the Swiss Armed Forces, this study helped to ensure the quality of this assessment and to identify possibilities for improvement. But in a broader perspective, however, the study may also provide guidance to researchers who have to deal with challenges similar to those that we faced (i.e., examining misfitting models for invariance in the context of unbalanced samples) and thus may help them to obtain robust findings despite these challenges.

Introduction

In the conscription system of the Swiss Armed Forces, cadre selection is based on various assessments. In an initial selection step, the training camp commander and his staff conduct interviews, military exercises, and knowledge tests with the recruits during the 18 to 21 weeks of basic military training. Recruits that are considered as most promising for a cadre position are sent to a cadre school after basic military training. The final assessments take place at the cadre school. There, it is decided which candidates are fit for lower cadre positions (i.e., squad leader positions) and thus will complete cadre school only and which candidates are fit for higher noncommissioned officer (NCO) or officer positions and thus are permitted entry into the higher NCO or officer school (Swiss Armed Forces, 2012).

In this final selection of NCOs and officers, psychological cadre assessment at the recruitment centers has a key role (Swiss Armed Forces, 2012). This includes the assessment of 19 leadership characteristics/dimensions that are grouped into the five broad sections: “leadership motivation,” “general mental ability,” “self-competence,” “social-competence,” and the category “extra dimensions” (see Table 1). Assessment of 13 of these 19 dimensions is based on the candidate’s self-ratings (i.e., motivation and personality), two dimensions on objective test results (i.e., general mental ability), and four dimensions on the candidate’s performance on a concept development exercise,¹ which is rated by two psychologists (Swiss Armed Forces, 2012). Scores on the 19 dimensions are then converted to stanine values, which is done to allow easier comparison of a candidate’s assessment result with the population norms of cadre applicants (Swiss Armed Forces, 2012). For the final cadre selection decision by the training camp commander and his staff, the stanine values are then summarized on a one-page output sheet (Swiss Armed Forces, 2012).

Table 1. Scales and their features and Cronbach’s alphas of the previously (i.e., 2012) and the presently studied (i.e., 2019) assessment cycles.

				Cronbach’s α – 2012 assessment cycle		Cronbach’s α – 2019 assessment cycle
Scale / construct	Items	Facets	Range	German(n = 1696)	French(n = 487)	German(n = 1284)	French(n = 474)	Reference
Leadership Motivation
Leadership motivation	4	1	-2-+2	.90	.89	.87	.89	Giauque, Vaso, & Boss (2006)
General mental ability
Matrices test	20	1	0-1	.74	.72	.71	.72	Hornke, Etzel, & Küppers (2000)
Concentration-stress test	200	4	0-1	.97M/.81A/.77O/.73C	.96M/.73A/.73O/.70C	.96M/.79A/.78O/.72C	.96M/.70A/.78O/.68C	Boss & Fischer (2006)
Concept (information processing)	4	1	1-4	-	-	-	-	Swiss Armed Forces (2012)
Self-competence
Achievement motivation	34	6	1-6	.97	.96	.96	.96	Gürber & Skupnjak (2006)
Stress tolerance	29	5	1-6	.83	.81	.82	.80	Stoll (2006)
Conscientiousness	10	1	1-6	.87	.83	.86	.79	Boss & Brenner (2006)
Independence	17	3	1-6	.89	.87	.87	.84	Melliger & Boss (2006)
Influencing behavior	23	4	1-6	.93	.93	.92	.92	Lécher & Boss (2006)
Concept (communication)	5	1	1-4	-	-	-	-	Swiss Armed Forces (2012)
Social-competence
Social behavior	22	4	1-6	.92	.89	.91	.88	Egger & Boss (2006a)
Conflict behavior	24	4	1-6	.88PR/.85PA	.86PR/.77PA	.87PR/.84PA	.83PR/.80PA	Egger & Boss (2006b)
Extraversion	10	1	1-6	.88	.87	.87	.85	Boss & Brenner (2006)
Agreeableness (courtesy)	10	1	1-6	.80	.78	.89	.89	Boss & Brenner (2006)
Agreeableness (cooperation)	10	1	1-6	.91	.90	.78	.78	Boss & Brenner (2006)
Concept (appearance)	2	1	1-4	-	-	-	-	Swiss Armed Forces (2012)
Concept (audience contact)	2	1	1-4	-	-	-	-	Swiss Armed Forces (2012)
Extra dimensions
Integrity	64	8	1-6	.94	.93	.93	.92	Egger & Boss (2006C)
Neuroticism	9	1	1-6	.80	.84	.79	.81	Boss & Brenner (2006)

Note. Scales selected for the purpose of the present study are displayed in gray. Items = number of scale items. Facets = number of facets / subtests that the scales are consisted of. Range = range of possible response values. For comparison purposes, Cronbach’s alphas are reported for two different assessment cycles: For the 2012 assessment cycle (see Goldammer, 2019) and for the 2019 assessment cycle (i.e., the cycle that is used in the present study). Cronbach’s alphas of concentration-stress test modules are abbreviated as follows: M = math test, A = analogy test, O = coordinate test, C = cutting pattern test. For the math, analogy, and coordinate tests, the reported Cronbach’s alphas values are based on both parts of each test. Cronbach’s alphas of the two conflict behavior facets are abbreviated as follows: PR = proactive facet, PA = passive facet.

As this psychological assessment takes place in Switzerland’s multilingual context, the employed measures not only need to be valid but also need to be equally valid across the language groups (e.g., German and French speakers). Regarding construct validity, for instance, not only do the items of the employed measures have to adequately reflect the underlying intended construct (i.e., the measurement model should fit the data); in addition, the employed measures need to have the same psychometric properties in each language group (i.e., the measurement model parameters should be invariant across groups).

Despite the relevance of this issue, no up-to-date and rigorous examination of measurement invariance of the psychological assessment is available, however. Also, previous studies on the psychometric properties of the measures (Goldammer, 2019; Maier, 2008) relied primarily on conventional model fit and invariance testing procedures (i.e., [single and multiple group] confirmatory factor analysis [CFA] with maximum likelihood [ML]), which offer limited options for dealing appropriately with models that fail the exact fit and/or exact invariance test (i.e., χ² and/or Δχ², respectively) (Muthén & Asparouhov, 2012, 2013).

The aim of this article is therefore to (re)examine selected general mental ability (GMA) and personality tests used in the psychological cadre assessment (rows in gray, Table 1) regarding model fit and measurement invariance across two major language groups in Switzerland (i.e., German and French speakers) by using Bayesian structural equation modeling (BSEM; Muthén & Asparouhov, 2012) – a modeling and invariance testing approach that takes potential model misfit and non-invariance more appropriately into account (e.g., Muthén & Asparouhov, 2012, 2013). In this study we focus on examining the most illustrative constructs of the test battery (i.e., general mental ability [concentration-stress test], conscientiousness, extraversion, agreeableness [cooperation], integrity, neuroticism), which are also commonly used by many other Armed Forces to assess the cognitive ability and personality of their personnel (e.g., Darr, 2011). By (re)examining the model fit and the measurement invariance of these constructs, this study therefore helps to maintain the high quality of the psychological cadre assessment of the Swiss Armed Forces and to improve it where necessary. In a broader perspective, the study may also provide guidance to researchers who have to deal with challenges similar to those that we faced (i.e., examining misfitting models for invariance in the context of unbalanced samples) and thus help them to obtain robust findings despite the challenges.

In the rest of this introduction, we first provide a brief overview of the theoretical background of the GMA and personality tests examined and describe the adaptations and new developments undertaken. We then describe the analytical framework applied (i.e., BSEM) in a little bit more detail. More specifically, we highlight the advantages of BSEM compared to traditional approaches (e.g., CFA with ML), when it comes to evaluation of model fit and measurement invariance.

Test development and content

All GMA and personality tests used in psychological cadre assessment are based on an in-depth theoretical background and well-established scales. However, to make the measures suitable for the military context and the target population (i.e., young adults aged 20), items of existing tests had to be adapted and some completely new items had to be developed (Boss & Brenner, 2006; Boss & Fischer, 2006; Egger & Boss, 2006c).

General mental ability

The largest GMA measure included in the psychological cadre assessment is the concentration-stress test, which is based on the Berlin Model of Intelligence Structure (BIS; Jäger, 1982, 1984) – a model that has considerable influence on intelligence measurement in German-speaking countries (e.g., Amthauer, Brocke, Liepmann, & Beauducel, 2001; Jäger, Süss, & Beauducel, 1997). The BIS distinguishes between two modalities (i.e., operations and contents). The four operations (i.e., processing speed, memory, creativity, processing capacity) and the three contents (i.e., verbal, numerical, figural) form a matrix of 12 unique tasks, which are hypothesized to indicate a general mental ability construct (Jäger, 1982, 1984). The four test modules that make up the concentration-stress test (i.e., numerological processing speed [math test]²; verbal reasoning [analogy test]; verbal and numerological memory [coordinate test]; figural reasoning [cutting pattern test]), reflect five to six cells of the BIS (Boss & Fischer, 2006). The concentration-stress test has a specific sequential order. At the start, the test taker must complete the first parts of the math, analogy, and memory tests. Next, the challenging tasks of the cutting pattern test have to be solved, which for most participants will induce the intended frustration, as only few participants will be able to rearrange the cutting patterns correctly. Afterward, the second parts of the math, analogy, and memory tests must be completed, to determine deviations from the results of the corresponding first parts and thus to infer the candidate’s consistency of performance or the lack of frustration tolerance (Boss & Fischer, 2006).

Personality

Another major part of the test battery assesses the candidate’s personality, which is largely based on Goldberg’s (1981) Big Five framework. For the development of the tailored Big Five scales used in the battery (e.g., conscientiousness, extraversion, agreeableness, and neuroticism), an item pool was first generated that included adapted items from existing Big Five or related inventories (e.g., NEO-FFI (Costa & McCrae, 1992); 16 PF (Cattell, Eber, & Tatsuoka, 1970)) and newly developed items (Boss & Brenner, 2006). Based on the results of three pretests with conscripts, this pool was then cleared of items that consistently showed an unclear factor loading pattern (i.e., low major loadings and/or high cross-loadings) and a low item-total correlation, which left 12 items per Big Five domain (Boss & Brenner, 2006). Based on the results of a subsequent study (Maier, 2008), the item set was further reduced. This resulted in the present item set of nine to 10 items per Big Five trait.

A similar procedure was applied when developing the integrity scale (Egger & Boss, 2006c). Based on three attitude-based facets of integrity (e.g., attitude toward rules and regulations) and five trait-based facets of integrity (e.g., self-control), which were considered as especially relevant for the present cadre selection context (Egger & Boss, 2006c), a large item pool was generated that contained adapted items from existing integrity inventories (e.g., Mussel, 2003) and newly developed items (Egger & Boss, 2006c). Expert ratings were then used to clear the pool of items with ambiguous wording, and based on a subsequently conducted study with conscripts, it was additionally cleared of items with an unclear factor loading pattern (i.e., low major loadings and/or high cross-loadings) and a low item-total correlation. With this reduction, eight items for each of the eight integrity facets remained in the final scale (Egger & Boss, 2006c).

Model-fit and invariance testing: From CFA with ML to BSEM

Once a newly developed test battery has reached a certain maturity, researchers normally move on from explorative stage and try to confirm the hypothesized measurement model (i.e., the factor structure) in new samples, as had been done in the development process of the Swiss Armed Forces’ test battery. In a conventional confirmatory measurement model set-up, selected parameters are freely estimated (e.g., major factor loadings), and other parameters are fixed to specific values (e.g., cross-loadings and residual correlations are fixed to zero) (Brown, 2015, pp. 35–87). This set-up of freely estimated and fixed parameters expresses the researchers’ hypotheses on how the latent (unobserved) factors are related to observed measures (i.e., test/questionnaire items) (Brown, 2015, p. 35–87). If the hypothesized structure should be also tested for equivalence across different groups (as needed in our case), the parameter set-up can be extended by including additional equality constraints for selected parameters (e.g., equality restrictions for factor loadings across groups) (Brown, 2015, pp. 206–286). In any case, the desired outcome of such a confirmatory test (i.e., CFA) is, of course, that the measurement model fits the data, or more technically, that the model-implied variance-covariance matrix matches the sample variance-covariance matrix (Brown, 2015, pp. 35–87).

Most often, however, the proposed measurement model(s) of the constructs used in the Swiss Armed Forces test battery turned out to fit the data unsatisfactorily (Goldammer, 2019; Maier, 2008), which means they failed the exact fit and/or exact invariance test (i.e., χ² and/or Δχ², respectively). Unfortunately, conventional CFA with ML mainly offers two limited (if not inappropriate) options to deal with such misfitting models (Muthén & Asparouhov, 2012, 2013). First, researchers can use approximate fit indices, such as CFI or RMSEA, to judge the fit of a model (e.g., Hu & Bentler, 1999), and the change of these indices, when examining measurement invariance (e.g., Cheung & Rensvold, 2002). Second, they can undertake a series of model modifications (e.g., adding residual correlations of freely estimating otherwise constrained parameters) based on the modification indices, to reach a well-fitting model. Both approaches seem unsatisfactory. The first approach, because it does not offer a remedy for the model misfit and/or non-invariance but rather an excuse to interpret the biased model estimates anyway, and the second approach, because it bears the risk of ending up with a model that does not generalize beyond the present sample (MacCallum, Roznowski, & Necowitz, 1992) or ending up with a model that does not entail the correct composition of invariant and non-invariant parameters across groups (Asparouhov & Muthén, 2014).

In contrast, BSEM is a newer modeling and invariance testing approach that can take the potential model misfit and non-invariance into account more appropriately (e.g., Muthén & Asparouhov, 2012, 2013). Let us take, for instance, the single-group case, in which the evaluation of measurement model fit is usually of primary interest. Compared to a conventional CFA with ML, in which strict assumptions for certain parameters are imposed (i.e., zero cross-loadings and residual correlations), BSEM allows the incorporation of prior beliefs for certain parameters in a more nuanced way. This could entail specifying a prior distribution with a mean of zero and a small variance for cross-loadings and/or residual correlations (see Asparouhov, Muthén, & Morin, 2015; Muthén & Asparouhov, 2012). The larger the prior variances on such parameters need to be to obtain a model that fits the data, the more flexible the model becomes and thus the weaker the theory becomes (Asparouhov et al., 2015). Thus, using BSEM offers two central benefits: First, it allows the researcher to keep on working with a measurement model that fits the data and nevertheless represents substantive theory. Second, it provides the researcher with a quantification of the strength of the underlying theory, in terms of the prior variance that is specified for the parameters (see Asparouhov et al., 2015).

In addition, BSEM also offers versatile modeling and analysis options for the multiple-group case, in which the evaluation of measurement invariance is usually of primary interest. Compared to a conventional multiple-group CFA with ML, in which exact equality constraints across groups are imposed for certain parameters (e.g., loadings and intercepts), BSEM allows the incorporation of the prior beliefs of the parameter invariance in a more nuanced way. This could entail, for instance, specifying a prior distribution with a mean of zero and a small variance for the difference between the factor loadings and/or indicator intercepts (Muthén & Asparouhov, 2013). The larger the prior variances on such difference parameters need to be to obtain a model that fits the data, the larger the magnitude of non-invariance (Muthén & Asparouhov, 2013). In the BSEM context, parameters are considered as significantly different from each other (i.e., non-invariant) if the 95% credibility interval of the difference between the parameters does not include zero. In addition, the BSEM approach of (approximate) invariance testing is also very attractive because it can be applied with models that cannot otherwise be tested for invariance (i.e., measurement models with a small variance prior on cross-loadings and/or residual-correlations).

BSEM therefore seems to be a far better modeling and analysis option than the conventional CFA with ML, especially if the hypothesized models fail the exact fit and/or invariance test. This was also the reason why we choose BSEM for the present investigation.

Methods

Sample

For this study we used the data from the complete assessment cycle in 2019, in which 1,843 cadre candidates completed the psychological assessment. The cadre candidates were on average 21.11 years old (SD = 1.81) and predominately men (1,776, or 96.4%). The sample consisted of 1,284 German-speaking, 474 French-speaking, and 85 Italian-speaking candidates. Thus, with exception of the language group, the sample was relatively homogeneous.

However, we decided to use only the larger two language groups (i.e., German and French speakers) for the present study. The reason for disregarding the Italian-speaking group was that we had planned to estimate models with up to 75 parameters in the single-group analyses and models with up to 152 parameters in the multiple-group analyses, which would have resulted in poor coverage of central estimates. For instance, the power to detect potential residual correlations within the Italian language group and parameter non-invariances between the Italian and the other language groups would have been below an acceptable minimum of .8 (see e.g., Muthén & Asparouhov, 2012).

The data set contained no missing values. For one, this is because non-responses in the cognitive tests are treated by default as incorrect responses. For another, this is because candidates have to provide a response for each item in order to proceed to the next survey page within the personality test part (i.e., forced-response format is used).

Study measures

General mental ability

The concentration-stress test (Boss & Fischer, 2006) is composed of four subtests that measure the GMA facets numerological processing speed (math test; two parts with 50 items each, 8-minute time limit for each part), verbal reasoning (analogy test; two parts with 20 items each, 6-minute time limit for each part), verbal and numerological memory (coordinate test; two parts with 10 items each, 7-minute time limit for each part), and figural reasoning (cutting pattern test; 20 items, 8-minute time limit). The language group-specific Cronbach’s alpha values for the subtests in the assessment cycle studied here (i.e., 2019 cycle) were the following α_German = .96, α_French = .96 for both parts of the math test, α_German = .79, α_French = .70 for both parts of the analogy test, α_German = .78, α_French = .78 for both parts of the coordinate test, and α_German = .72, α_French = .68 for the cutting pattern test. At least at first sight (i.e., without conducting any formal test), the language-group specific alpha values of the presently studied cycle appeared to be fairly similar to those that were reported for the previously studied assessment cycle (i.e., column Cronbach’s α – 2012 cycle, Table 1).

Personality

The candidates rated all items of the Big five scales (Boss & Brenner, 2006) and the integrity scale (Egger & Boss, 2006c) on a Likert scale ranging from 1 = completely disagree to 6 = completely agree. For each of the five personality scales, the following provides a sample item and the corresponding language-group specific Cronbach’s alpha values for the presently studied assessment cycle (i.e., 2019 cycle): “I like to work very precisely” (conscientiousness; 10 items; α_German = .86, α_French = .79), “I can make new friends quickly” (extraversion; 10 items; α_German = .87, α_French = .85), “I prefer tasks that I can do together with work colleagues” (agreeableness[cooperation]; 10 items; α_German = .78, α_French = .78), “I take honesty more seriously than many others” (integrity; 64 items; α_German = .93, α_French = .92), “There are times when I feel absolutely worthless” (neuroticism; 9 items; α_German = .79, α_French = .81). With exception of the agreeableness [cooperation] scale, for which the language group-specific alpha values tended to be lower, the language-group specific alpha values of the other four personality scales in the presently studied cycle appeared to be fairly similar (i.e., without conducting any formal test) to those that were reported for the previously studied assessment cycle (i.e., column Cronbach’s α – 2012 assessment cycle, Table 1).

Analytical procedure

Measurement model evaluation

To evaluate the measurement model fit of the constructs, we used BSEM (Muthén & Asparouhov, 2012) in Mplus Version 8.4 (Muthén & Muthén, 1998–2017). A series of BSEM models were estimated for each construct.³ For identification purposes, the factor variance was fixed to 1 within these models. Moreover, factor indicators were standardized so that the prior variance had similar interpretation across items. First, a measurement model with non-informative priors for the major loadings and exact zero residual correlations were estimated. Estimating a BSEM model with non-informative priors for the major loadings and exact zero residual correlations is comparable to estimating a conventional CFA with ML (Muthén & Asparouhov, 2012). This restricted BSEM model then served as baseline model for more liberal BSEM models in which approximate zero residual correlations with increasingly larger prior variances were specified. In this comparison process, which is also referred to as sensitivity analysis (Asparouhov et al., 2015), we examined the fit of models with five different prior variances ranging from 0.0001 to 0.01. With a standardized factor variance and standardized indicators, a prior variance of 0.01, for instance, implies a prior belief that 95% of the distribution of a residual correlation will fall within a range between ± .2 (Muthén & Asparouhov, 2012). All these analyses were conducted separately for the German-speaking and the French-speaking group.

The models were estimated with the Markov chain Monte Carlo (MCMC) algorithm using the Gibbs sampler and two chains, each of which was run with 50,000 iterations. The convergence of the estimation process was assessed with the potential scale reduction factor (PSRF). In this case, PSRF values lower than 1.10 were taken as an indication of model convergence (Gelman, Carlin, Stern, & Rubin, 2004, p. 287). All models listed in Table 2 reached convergence according to this criterion.

Table 2. Model fit results of models with different levels of residual correlations.

Model	Pars.	PPP	95% CI	DIC	BCFI(90%CI)	BRMSEA(90%CI)
General mental ability (concentration-stress test) (1 first order factor, 4 subtest scores as indicators)
German
Exact zero residual correlations	12	.035	- 1.13 – 27.85	14000.60	.979(.963 – .989)	.071(.051 – .094)
Approx. zero residual correlations (v < .0001)	18	.442	- 13.76 – 17.29	13988.21	.999(.985 – 1)	.013(0 – .064)
Approx. zero residual correlations (v = .0001)	18	.479	- 14.21 – 15.92	13987.05	1(.987 – 1)	0(0 – .052)
Approx. zero residual correlations (v = .0003)	18	.495	- 14.53 – 15.38	13984.13	1(.987 – 1)	0(0 – .037)
Approx. zero residual correlations (v = .001)	18	.502	- 14.83 – 15.27	13981.02	1(.988 – 1)	0(0 – .028)
Approx. zero residual correlations (v = .01)	18	.506	- 14.76 – 15.19	13966.53	1(.988 – 1)	0(0 – .016)
French
Exact zero residual correlations	12	.359	- 12.11 – 17.11	5177.95	.991(.957 – 1)	.046(0 – .104)
Approx. zero residual correlations (v < .0001)	18	.345	- 13.05 – 20.53	5179.61	.988(.946 – 1)	.072(0 – .158)
Approx. zero residual correlations (v = .0001)	18	.423	- 14.14 – 17.85	5177.74	.995(.957 – 1)	.042(0 – .137)
Approx. zero residual correlations (v = .0003)	18	.466	- 14.72 – 16.04	5176.37	1(.964 – 1)	.005(0 – .109)
Approx. zero residual correlations (v = .001)	18	.484	- 14.83 – 15.60	5175.37	1(.966 – 1)	0(0 – .090)
Approx. zero residual correlations (v = .01)	18	.492	- 15.03 – 14.81	5171.28	1(.968 – 1)	0(0 – .053)
Conscientiousness (1 first order factor, 10 items as indicators)
German
Exact zero residual correlations	30	.000	338.13 – 392.16	32713.77	.912(.909 – .915)	.090(.088 – .092)
Approx. zero residual correlations (v < .0001)	75	.400	- 27.57 – 36.74	32386.99	.999(.995 – 1)	.053(0 – .136)
Approx. zero residual correlations (v = .0001)	75	.447	- 29.13 – 34.33	32384.65	1(.996 – 1)	.035(0 – .128)
Approx. zero residual correlations (v = .0003)	75	.480	- 30.75 – 32.31	32383.02	1(.996 – 1)	.016(0 – .122)
Approx. zero residual correlations (v = .001)	75	.491	- 30.56 – 31.57	32382.43	1(.996 – 1)	0(0 – .121)
Approx. zero residual correlations (v = .01)	75	.491	- 30.71 – 32.11	32382.49	1(.996 – 1)	0(0 – .122)
French
Exact zero residual correlations	30	.000	74.53 – 129.20	12534.55	.904(.891 – .915)	.079(.074 – .084)
Approx. zero residual correlations (v < .0001)	75	.254	- 23.10 – 46.13	12479.23	.989(.972 – 1)	.155(0 – .264)
Approx. zero residual correlations (v = .0001)	75	.368	- 28.96 – 39.04	12472.65	.995(.979 – 1)	.108(0 – .230)
Approx. zero residual correlations (v = .0003)	75	.442	- 30.80 – 35.62	12469.33	.998(.983 – 1)	.070(0 – .213)
Approx. zero residual correlations (v = .001)	75	.489	- 30.69 – 33.62	12466.85	.999(.985 – 1)	.039(0 – .202)
Approx. zero residual correlations (v = .01)	75	.510	- 31.85 – 30.45	12465.25	1(.986 – 1)	0(0 – .197)
Extraversion (1 first order factor, 10 items as indicators)
German
Exact zero residual correlations	30	.000	644.39 – 698.80	32191.84	.865(.862 – .867)	.122(.121 – .123)
Approx. zero residual correlations (v < .0001)	75	.402	- 27.88 – 37.15	31558.49	.999(.996 – 1)	.053(0 – .138)
Approx. zero residual correlations (v = .0001)	75	.450	- 29.37 – 34.49	31556.16	1(.996 – 1)	.036(0 – .128)
Approx. zero residual correlations (v = .0003)	75	.476	- 30.84 – 32.70	31554.56	1(.997 – 1)	.014(0 – .119)
Approx. zero residual correlations (v = .001)	75	.491	- 30.44 – 32.42	31.554.02	1(.997 – 1)	0(0 – .118)
Approx. zero residual correlations (v = .01)	75	.491	- 31.33 – 31.91	31553.90	1(.997 – 1)	0(0 – .120)
French
Exact zero residual correlations	30	.000	282.08 – 336.76	12090.93	.820(.812 – .827)	.137(.134 – .139)
Approx. zero residual correlations (v < .0001)	75	.265	- 23.08 – 45.33	11828.33	.994(.983 – 1)	.152(0 – .263)
Approx. zero residual correlations (v = .0001)	75	.373	- 28.21 – 38.61	11821.77	.997(.987 – 1)	.107(0 – .230)
Approx. zero residual correlations (v = .0003)	75	.445	- 30.27 – 35.55	11818.71	.999(.989 – 1)	.069(0 – .217)
Approx. zero residual correlations (v = .001)	75	.481	- 30.30 – 33.48	11816.33	1(.991 – 1)	.042(0 – .208)
Approx. zero residual correlations (v = .01)	75	.515	- 31.56 – 30.91	11814.34	1(.992 – 1)	0(0 – .193)
Agreeableness (cooperation) (1 first order factor, 10 items as indicators)
German
Exact zero residual correlations	30	.000	1023.38 – 1077.71	31359.95	.830(.828 – .832)	.153(.152 – .154)
Approx. zero residual correlations (v < .0001)	75	.404	- 27.82 – 36.66	30347.35	.999(.997 – 1)	.052(0 – .140)
Approx. zero residual correlations (v = .0001)	75	.452	- 29.74 – 34.20	30344.96	1(.997 – 1)	.033(0 – .124)
Approx. zero residual correlations (v = .0003)	75	.485	- 31.01 – 32.55	30343.34	1(.997 – 1)	.001(0 – .118)
Approx. zero residual correlations (v = .001)	75	.495	- 30.70 – 32.40	30342.96	1(.998 – 1)	0(0 – .119)
Approx. zero residual correlations (v = .01)	75	.489	- 30.35 – 31.42	30342.71	1(.998 – 1)	0(0 – .124)
French
Exact zero residual correlations	30	.000	395.71 – 449.71	11610.16	.817(.812 – .822)	.160(.158 – .162)
Approx. zero residual correlations (v < .0001)	75	.268	- 23.70 – 45.11	11234.96	.995(.988 – 1)	.151(0 – .260)
Approx. zero residual correlations (v = .0001)	75	.378	- 27.92 – 38.18	11227.81	.998(.991 – 1)	.105(0 – .232)
Approx. zero residual correlations (v = .0003)	75	.450	- 30.85 – 35.96	11224.66	.999(.992 – 1)	.066(0 – .212)
Approx. zero residual correlations (v = .001)	75	.494	- 30.82 – 33.72	11222.45	1(.993 – 1)	.037(0 – .206)
Approx. zero residual correlations (v = .01)	75	.515	- 32.50 – 30.46	11220.65	1(.994 – 1)	0(0 – .187)
Integrity (1 first order factor, 8 subtest scores as indicators)
German
Exact zero residual correlations	24	.000	299.90 – 345.11	25406.08	.922(.919 – .924)	.112(.110 – .114)
Approx. zero residual correlations (v < .0001)	52	.406	- 23.93 – 30.92	25106.30	.999(.996 – 1)	.046(0 – .122)
Approx. zero residual correlations (v = .0001)	52	.453	- 24.83 – 28.83	25104.44	1(.996 – 1)	.028(0 – .117)
Approx. zero residual correlations (v = .0003)	52	.482	- 25.60 – 26.90	25103.07	1(.997 – 1)	.008(0 – .109)
Approx. zero residual correlations (v = .001)	52	.497	- 26.39 – 26.61	25102.31	1(.997 – 1)	0(0 – .109)
Approx. zero residual correlations (v = .01)	52	.501	- 26.39 – 26.43	25100.21	1(.997 – 1)	0(0 – .064)
French
Exact zero residual correlations	24	.000	61.79 – 108.68	9588.77	.934(.925 – .943)	.095(.089 – .101)
Approx. zero residual correlations (v < .0001)	52	.284	- 19.38 – 38.39	9532.22	.994(.982 – 1)	.131(0 – .237)
Approx. zero residual correlations (v = .0001)	52	.377	- 23.49 – 32.12	9527.72	.997(.986 – 1)	.091(0 – .212)
Approx. zero residual correlations (v = .0003)	52	.457	- 25.47 – 29.53	9524.32	.999(.989 – 1)	.042(0 – .193)
Approx. zero residual correlations (v = .001)	52	.483	- 25.91 – 27.68	9523.09	1(.990 – 1)	.014(0 – .185)
Approx. zero residual correlations (v = .01)	52	.494	- 26.85 – 26.92	9522.47	1(.990 – 1)	0(0 – .181)
Neuroticism (1 first order factor, 9 items as indicators)
German
Exact zero residual correlations	27	.000	146.49 – 196.89	30630.67	.928(.923 – .933)	.071(.068 – .073)
Approx. zero residual correlations (v < .0001)	63	.404	- 26.61 – 33.97	30488.00	.999(.992 – 1)	.050(0 – .132)
Approx. zero residual correlations (v = .0001)	63	.455	- 27.74 – 31.46	30485.76	.999(.993 – 1)	.032(0 – .122)
Approx. zero residual correlations (v = .0003)	63	.490	- 29.07 – 29.39	30484.07	1(.994 – 1)	.004(0 – .098)
Approx. zero residual correlations (v = .001)	63	.501	- 29.40 – 28.92	30483.25	1(.994 – 1)	0(0 – .087)
Approx. zero residual correlations (v = .01)	63	.508	- 29.76 – 28.23	30482.10	1(.994 – 1)	0(0 – .070)
French
Exact zero residual correlations	27	.000	56.38 – 106.00	11105.62	.928(.916 – .938)	.079(.074 – .085)
Approx. zero residual correlations (v < .0001)	63	.262	- 21.39 – 42.33	11062.45	.991(.976 – 1)	.143(0 – .256)
Approx. zero residual correlations (v = .0001)	63	.363	- 25.08 – 36.20	11057.03	.996(.982 – 1)	.103(0 – .225)
Approx. zero residual correlations (v = .0003)	63	.455	- 27.61 – 31.73	11053.19	.998(.985 – 1)	.060(0 – .201)
Approx. zero residual correlations (v = .001)	63	.481	- 27.88 – 30.05	11051.79	1(.987 – 1)	.030(0 – .196)
Approx. zero residual correlations (v = .01)	63	.500	- 28.89 – 29.29	11050.61	1(.988 – 1)	0(0 – .193)

Note. Factor variance was fixed to 1 for identification purposes. Factor indicators were standardized so that the prior variance has similar interpretation across items. Pars. = estimated model parameters; PPP = posterior predictive p-value; 95% CI = 95% confidence interval for the difference between the observed and the replicated model χ² values; DIC = deviance information criterion; BCFI = Bayesian comparative fit index; BRMSEA = Bayesian root mean square error of approximation; For BCFI and BRMSEA, the median is reported as point estimate with a highest density 90% credibility interval. v = prior variance for the approximate zero residual correlations. Models considered for further investigation are shown in gray.

The fit of the measurement models was assessed with posterior predictive checking and the associated posterior predictive (PP) p-value. Whereas a PP p-value around .50 is indicative of an excellent-fitting model (Muthén & Asparouhov, 2012), PP p-values above .10 or .15 have been taken as indicating a reasonable fit (Cain & Zhang, 2019). In addition, the deviance information criterion (DIC) was used for model comparisons within each construct. In this case, lower DIC values were indicative of a better fitting model (Asparouhov et al., 2015).

The final measurement model for each construct was selected based on whether it offered an adequate balance between model fit (i.e., of two models with different priors, the model that exhibited a better fit to the data was favored) and model parsimony (i.e., if two models fitted equally, the one with smaller prior was favored over the model with larger prior), and on whether the model estimates were comparable and in the expected range in both language groups (i.e., we favored models that had the same configuration of major loadings and residual correlations in the groups over models that had a different configuration of major loadings and residual correlations in the groups).

Measurement invariance evaluation

The final measurement models of the constructs were then further tested for invariance across the language groups.⁴ For this evaluation, we again used the BSEM modeling and testing framework (Muthén & Asparouhov, 2013).

A series of scalar invariance models (i.e., models in which invariance of loadings and intercepts are imposed) with language group-specific residuals (co)variances were estimated for each construct. These models were identified by setting the factor variance and the factor mean in the French-speaking group to 1 and 0, respectively. We focused on testing for scalar invariance, as this type of invariance is sufficient for most structural comparisons across groups (e.g., latent means)⁵ (Steenkamp & Baumgartner, 1998). However, if the focus was on comparing observed composite scores, an additional test for (approximate) invariance of residual (co)variances would be necessary (Steenkamp & Baumgartner, 1998). First, a measurement model with exact scalar invariance was estimated. This restricted BSEM model then served as a baseline model for more liberal BSEM models in which approximate scalar invariance with increasingly larger prior variances were specified. In this sensitivity analysis (Asparouhov et al., 2015), we examined the fit of models with five different prior variances ranging from 0.001 to 0.1 (see Muthén & Asparouhov, 2013; Winter & Depaoli, 2020). With a standardized factor variance and standardized indicators, a prior variance of 0.1, for instance, implies a prior belief that 95% of the distribution of the difference/non-invariance will fall into a range between ± .62 (Muthén & Asparouhov, 2013). Note, however, that we performed the invariance tests based on the unstandardized model estimates, as the use of standardized estimates would have implied the unlikely assumption of equal indicator variances across groups.

In this measurement invariance testing, the same estimation procedures were used, and the same criteria were applied when examining convergence (all models listed in Table 3 achieved convergence according to the PSRF criterion) and model fit as in the previous evaluation of the measurement models.

Table 3. Model fit results of models with different levels of scalar invariance across language groups.

Model	Pars.	PPP	95% CI	DIC	BCFI(90%CI)	BRMSEA(90%CI)
General mental ability (concentration-stress test)(Model with approximate zero residual covariances with a prior variance of 6.25*10^−6)^a
Exact scalar invariance	30	.103	-8.65 – 42.23	-7671.53	.981(.963 – 1)	.010(0 – .015)
Approx. scalar invariance (v = .001)	38	.247	-14.79 – 31.74	-7517.72	.991(.976 – 1)	.030(0 – .053)
Approx. scalar invariance (v = .005)	38	.249	-14.68 – 31.43	-7831.70	.991(.976 – 1)	.005(0 – .009)
Approx. scalar invariance (v = .01)	38	.249	-14.70 – 31.60	-8371.04	.991(.976 – 1)	.003(0 – .005)
Approx. scalar invariance (v = .05)	38	.246	-14.51 – 31.96	-8893.27	.991(.976 – 1)	.002(0 – .004)
Approx. scalar invariance (v = .1)	38	.245	-14.47 – 32.07	-8861.40	.991(.976 – 1)	.003(0 – .004)
Conscientiousness(Model with approximate zero residual covariances with a prior variance of .001)
Exact scalar invariance	132	.427	-41.54 – 50.73	45711.80	.999(.995 – 1)	.063(0 – .165)
Approx. scalar invariance (v = .001)	152	.456	-42.03 – 46.59	45707.72	1(.995 – 1)	.033(0 – .116)
Approx. scalar invariance (v = .005)	152	.132	-21.74 – 73.74	45730.50	.995(.990 – 1)	.083(0 – .119)
Approx. scalar invariance (v = .01)	152	.321	-36.74 – 55.52	45715.12	.998(.994 – 1)	.062(0 – .116)
Approx. scalar invariance (v = .05)	152	.483	-44.07 – 43.44	45706.06	1(.996 – 1)	.023(0 – .105)
Approx. scalar invariance (v = .1)	152	.486	-44.16 – 45.11	45705.16	1(.996 – 1)	.016(0 – .097)
Extraversion(Model with approximate zero residual covariances with a prior variance of .001)
Exact scalar invariance	132	.439	-42.58 – 50.77	48422.83	.999(.996 – 1)	.060(0 – .170)
Approx. scalar invariance (v = .001)	152	.458	-4288 – 46.71	48420.88	1(.996 – 1)	.041(0 – .134)
Approx. scalar invariance (v = .005)	152	.461	-43.37 – 47.12	48419.31	1(.996 – 1)	.029(0 – .098)
Approx. scalar invariance (v = .01)	152	.294	-33.63 – 58.50	48429.43	.998(.995 – 1)	.063(0 – .110)
Approx. scalar invariance (v = .05)	152	.481	-43.94 – 43.57	48419.20	1(.997 – 1)	.026(0 – .127)
Approx. scalar invariance (v = .1)	152	.484	-45.18 – 44.86	48418.25	1(.997 – 1)	.020(0 – .108)
Agreeableness (cooperation)(Model with approximate zero residual covariances with a prior variance of .01)
Exact scalar invariance	132	.488	-44.81 – 45.61	44488.78	1(.997 – 1)	.015(0 – .108)
Approx. scalar invariance (v = .001)	152	.479	-44.84 – 53.53	44495.19	1(.997 – 1)	.040(0 – .180)
Approx. scalar invariance (v = .005)	152	.467	-44.59 – 56.41	44473.28	1(.997 – 1)	.009(0 – .042)
Approx. scalar invariance (v = .01)	152	.304	-33.93 – 58.71	44499.92	.999(.996 – 1)	.065(0 – .120)
Approx. scalar invariance (v = .05)	152	.503	-44.46 – 44.89	44489.16	1(.997 – 1)	.009(0 – .130)
Approx. scalar invariance (v = .1)	152	.513	-44.87 – 43.93	44488.40	1(.997 – 1)	0(0 – .118)
Integrity(Model with approximate zero residual covariances with a prior variance of .01)
Exact scalar invariance	90	.458	-36.40 – 41.08	26000.22	1(.996 – 1)	.041(0 – .149)
Approx. scalar invariance (v = .001)	106	.162	-19.38 – 59.34	26011.23	.996(.993 – 1)	.053(0 – .080)
Approx. scalar invariance (v = .005)	106	.482	-36.22 – 38.51	25997.36	1(.996 – 1)	.015(0 – .096)
Approx. scalar invariance (v = .01)	106	.509	-37.17 – 37.12	25996.36	1(.997 – 1)	0(0 – .098)
Approx. scalar invariance (v = .05)	106	.514	-37.45 – 36.80	25994.15	1(.997 – 1)	0(0 – .073)
Approx. scalar invariance (v = .1)	106	.511	-37.49 – 36.97	25991.88	1(.997 – 1)	0(0 – .060)
Neuroticism(Model with approximate zero residual covariances with a prior variance of .001)
Exact scalar invariance	110	.460	-39.04 – 43.79	44650.40	.999(.994 – 1)	.049(0 – .172)
Approx. scalar invariance (v = .001)	128	.461	-39.40 – 44.92	44650.32	.999(.994 – 1)	.049(0 – .177)
Approx. scalar invariance (v = .005)	128	.458	-39.13 – 43.89	44648.82	.999(.994 – 1)	.033(0 – .122)
Approx. scalar invariance (v = .01)	128	.453	-39.56 – 44.72	44646.85	.999(.993 – 1)	.024(0 – .084)
Approx. scalar invariance (v = .05)	128	.484	-39.98 – 44.26	44647.25	1(.994 – 1)	.021(0 – .103)
Approx. scalar invariance (v = .1)	128	.498	-40.40 – 43.05	44645.65	1(.994 – 1)	.013(0 – .084)

Note. For identification purposes, the factor variance and factor mean were fixed in the French-speaking group to 1 and 0, respectively. In all scalar invariance models, the residual (co)variances were freely estimated across the language groups. Unstandardized model estimates were used for comparisons across groups. Subtest scores were used as indicators for the models of general mental ability (concentration-stress test) and integrity. Pars. = estimated model parameters; PPP = posterior predictive p-value; 95% CI = 95% confidence interval for the difference between the observed and the replicated model χ² values; DIC = deviance information criterion; BCFI = Bayesian comparative fit index; BRMSEA = Bayesian root mean square error of approximation; For BCFI and BRMSEA, the median is reported as point estimate with a highest density 90% credibility interval. v = prior variance for the difference between the measurement parameters (i.e., loadings and intercepts) of the German language group and those of the French language group. Models considered for further investigation are shown in gray.

^a= The unstandardized variances of the four indicators of the concentration-stress were remarkably different from 1 (i.e., on average only 0.0265). Thus, a smaller prior variance had to be chosen than for the single-group analyses (see also footnote 5). In order to ensure the same ratio between the indicator variance (i.e., 1) and the prior variance for the residual covariances (i.e., .0003) as we had observed it in the single-group analysis, we aimed for a prior variance of 7.95*10⁻⁶ in the unstandardized case. The inverse Wishart (IW) setting of IW(0,60) with a prior variance of 6.25*10⁻⁶ approached this value best. In addition, runs with larger IW settings (i.e., [0,65], [0,75], [0,85]) resulted in only marginal improvement of model fit. Thus IW(0,60) was selected as prior variance for the residual covariances in the multiple-group analyses of the concentration-stress test.

The final scalar invariance model for each construct was selected based on whether it offered an adequate balance between model fit (i.e., of two models with different priors, the model that exhibited a better fit to the data was favored) and model parsimony (i.e., if two models fitted equally, the one with smaller prior was favored over the model with larger prior). In addition, we also checked for the plausibility of the estimates and favored well-fitting models with plausible estimates over well-fitting models with implausible estimates (e.g., models in which loadings turned out to be negative and/or non-significant, and/or residual covariances were very large and significant).

Subsampling

Since the sample sizes were not equal across the language groups, we additionally ran subsample comparisons based on the final scalar invariance model (Yoon & Lai, 2018) as a robustness check. If measurement invariance is tested in the context of unbalanced sample sizes, the power to detect a true non-invariance may be diminished (Kaplan & George, 1995; Yoon & Lai, 2018), because larger groups have a stronger influence than smaller groups on the joint fit function of the corresponding multiple group analysis (Brown, 2015, p. 251; Yoon & Lai, 2018, p. 203). In other words, a misfit (i.e., non-invariance) may not be detected, because the model is estimated in favor of the larger group (i.e., it fits for the majority but not for the minority). As a remedy, Yoon and Lai (2018) therefore proposed repeating the invariance test across multiple randomly drawn subsamples in which the comparison groups are of equal size.⁶

For the subsample comparisons, we generated 100 data sets of which each contained 474 randomly drawn participants from the German-speaking sample (n =1,284) and the 474 participants from the French-speaking sample. In addition to the non-invariance determined in the full-sample comparisons, we considered a parameter difference as potentially substantial (i.e., worth further investigation) if the difference between the parameters turned out to be significant in a substantial proportion (i.e., we considered 20% as substantial) of the subsample comparisons.

Results

Descriptive statistics

The language-group specific correlation matrices as well as the means and standard deviations of the construct indicators are reported in the supplementary material, Excel file “Descriptive statistics.”

Measurement model fit

Table 2 provides a summary of model fit for models, in which exact or different levels of approximate zero residual correlations were specified. The results in Table 2 are reported per construct and separately for each language group.

For all constructs, the strictest BSEM model (i.e., the model with exact zero residual correlations) provided a bad fit to the data, not only according to the PPP but also if alternative fit indices (i.e., comparative fit index [CFI; Bentler, 1990] or root mean square error of approximation [RMSEA; Steiger & Lind, 1980]) had been used for model fit evaluation. Thus, if we had insisted on using the model specifications that are conventionally applied in a CFA with ML, our examination of measurement invariance of the constructs would have been over before it even started, as the basic prerequisite of a well-fitting measurement model was not met for the examined constructs (see Steenkamp & Baumgartner, 1998, p. 80). In contrast, a well-fitting measurement model could be obtained for all constructs and in both language groups if more liberal BSEM models were estimated that allowed for a small prior variance of the residual correlations.

Take the model fit evaluation of the extraversion construct, for instance. Even the specification of a tiny prior variance of (v < .0001) for the residual correlations resulted in an model that showed a noticeably better fit to the data (i.e., PPP_German = .402, PPP_French = .265), than the model in which exact zero residual correlations were specified (i.e., PPP_German = .000, PPP_French = .000). Thus, if “fit to the data” had been the only criterion by which we judged the suitability of a model, the specification of a tiny prior variance (e.g., v < .0001) would have been sufficient for the extraversion and also all other constructs. In addition to the model fit, however, we also aimed for models that had the same configuration of major loadings and residual correlations in the language groups, in order to have an optimal basis for the following invariance testing. In the case of the extraversion construct, for instance, models with smaller prior variances (i.e., v < .0001, v = .0001, v = .0003) on the residual correlations resulted in different sets of significant residual correlations across the language groups. In contrast, in the model with a prior variance of .001, only two residual correlations turned out to be significant. One of them occurred in both language groups and the other one uniquely in the French-speaking group (for details, see supplementary material, Excel file “Model testing summary”). Therefore, the model with a prior variance of .001 was selected as final BSEM model for the extraversion construct. The same principle was applied when selecting the final BSEM models of the other constructs.

In all the final BSEM models (rows in gray, Table 2) all loadings were significant and sufficiently large (i.e., except for five loadings, all of them were larger than .4) (Brown, 2015). In addition, the vast majority of the freely estimated residual correlations in these final models turned out to be close to zero and non-significant (for details, see supplementary material, Excel file “Model testing summary”). In the rare case in which a residual correlation escaped the prior variance and turned out to be significant despite the wiggle room that we allowed (i.e., small prior variance for the parameter), the involved items were flagged to allow closer examination of their content. This examination revealed that significant residual correlations were most likely due to similar item wording.

In general, the results therefore indicated that the misfit of the strict BSEM models was mostly due to many small unmodeled residual correlations, which, however, are of little practical importance and thus do not call into question the general model specification (see Asparouhov et al., 2015). We therefore concluded that the final measurement models offered an adequate representation of the (co)variances in the data and consequently could be used to test for measurement invariance across language groups.

Measurement invariance

Table 3 provides a summary of model fit for models in which exact or different levels of approximate scalar invariance across the language groups were specified. The results in Table 3 are listed separately for each construct.

For all constructs, liberal BSEM models (i.e., models with approximate scalar invariance) offered a better representation of the (non)invariance in the measurement models than the strictest BSEM model (i.e., model with exact scalar invariance), not only in terms of model fit but also in terms of the plausibility of the model estimates.

Again, take the extraversion construct as an example. When forcing the loadings and intercepts of this construct to be exactly equal across groups, the inequality of these parameters had to seemingly channel through other unconstrained parameters, such as the residual (co)variances, which became inflated and significant. Similar effects (i.e., inflated significant residual correlations) occurred for approximate scalar invariance models in which only a small prior variance for the difference parameters was specified (i.e., v = .001, v = .005). This stands in contrast to a regular pattern of loadings and residual (co)variances (i.e., significant loadings and non-significant residuals covariances to those observed in the single-group analyses) that could be observed for the model in which a prior variance of .1 was specified for the difference parameters (for details, see supplementary material, Excel file “Invariance testing summary”). Therefore, the model with the prior variance of .1 was selected as final approximate scalar invariance model of the extraversion construct. Notably, the pattern that we observed for the extraversion construct (i.e., implausible estimates for models in which no or only small prior variances for the difference parameters were specified) also occurred when we examined the other constructs. For selecting the final approximate scalar invariance models of the other constructs (rows in gray, Table 3), we applied the same principle as in the case of the extraversion construct (i.e., we favored well-fitting models with plausible estimates over well-fitting models with implausible estimates).

These final approximate scalar invariance models were then further examined for significant loading and intercept differences (i.e., non-invariances). First, we examined the parameter differences in the full sample. Table 4 (column “Full sample comparisons”) shows the loading and intercept estimates of the construct extraversion for both language groups, as well as the differences of the parameters.⁷ The results of the full sample comparisons of the other constructs are available as supplementary material (see supplementary material, Tables S1-S5, columns “Full sample comparisons”).

Table 4. Parameter difference tests for the extraversion construct across the full sample and the subsamples.

	Full sample comparisons
	German (n = 1,284)		French (n = 474)			Subsample comparisons (k = 100[n = 474 ; n = 474])
Parameters	Estimate	SD	Estimate	SD	Diff(95% CI)	MDiff	SDDiff	% sig Diff
Loadings
λ1	0.909	0.141	0.937	0.108	-0.023(-0.346 – 0.284)	-0.027	0.025	0
λ2	0.645	0.123	0.570	0.070	0.079(-0.181 – 0.337)	0.077	0.032	0
λ3	0.943	0.150	0.805	0.105	0.140(-0.192 – 0.479)	0.157	0.026	0
λ4	0.806	0.117	0.771	0.073	0.037(-0.214 – 0.287)	0.030	0.026	0
λ5	0.897	0.137	0.835	0.098	0.070(-0.232 – 0.369)	0.076	0.034	0
λ6	0.597	0.095	0.503	0.059	0.096(-0.113 – 0.304)	0.095	0.026	0
λ7	0.618	0.136	0.487	0.142	0.140(-0.226 – 0.490)	0.140	0.036	0
λ8	0.992	0.151	0.785	0.080	0.210(-0.099 – 0.516)	0.217	0.024	0
λ9	0.637	0.129	0.687	0.130	-0.036(-0.356 – 0.284)	-0.031	0.035	0
λ10	0.905	0.152	1.066	0.102	-0.156(-0.470 – 0.186)	-0.133	0.030	0
Intercepts
τ1	4.776	0.130	4.686	0.061	0.091(-0.156 – 0.349)	0.068	0.030	0
τ2	4.598	0.094	5.077	0.042	-0.478*(-0.653 – -0.284)	-0.485	0.029	100
τ3	3.873	0.133	4.090	0.057	-0.217(-0.471 – 0.047)	-0.230	0.032	9
τ4	4.967	0.114	4.917	0.049	0.049(-0.166 – 0.277)	0.034	0.019	0
τ5	4.191	0.128	3.995	0.056	0.196(-0.051 – 0.450)	0.174	0.028	0
τ6	5.099	0.085	5.276	0.037	-0.177*(-0.341 – -0.009)	-0.190	0.024	79
τ7	4.257	0.091	3.800	0.056	0.458*(0.264 – 0.656)	0.438	0.034	100
τ8	4.401	0.140	4.661	0.051	-0.261(-0.521 – 0.013)	-0.276	0.029	52
τ9	4.988	0.095	4.574	0.058	0.415*(0.221 – 0.620)	0.398	0.030	100
τ10	4.590	0.126	4.524	0.064	0.063(-0.184 – 0.304)	0.053	0.029	0

Note. The full sample and the subsample comparisons were based on the approximate scalar invariance model (with a prior variance of .1) with approximate zero residual covariances (with a prior variance of .001). The residual (co)variances were freely estimated across the language groups. For identification purposes, the factor variance and factor mean were fixed in the French-speaking group to 1 and 0, respectively. Estimate = unstandardized posterior point estimate (i.e., median); SD = standard deviation of the estimate’s posterior distribution; Diff_{(95% CI)} = posterior difference estimate with 95% highest density credibility interval. k = number of subsamples (i.e., 100); M_Diff = average parameter difference across the subsample comparisons; SD_Diff = standard deviation of the parameter difference across the subsample comparisons; % sig Diff = percentage of subsample comparisons in which the parameter difference was significantly larger than zero (i.e., 95% highest density credibility interval did not include zero).

* = 95% highest density credibility interval of the parameter difference does not include zero.

The row with the additional potential non-invariant parameter across the language groups (i.e., τ₈) that was detected through the subsampling comparisons is highlighted in gray.

When examining the loadings of the extraversion construct across the language groups, none of the comparisons revealed a significant difference. Thus, full approximate metric (i.e., loading) invariance may be claimed for the extraversion items. However, four intercepts (i.e., τ₂, τ₆, τ₇, τ₈) turned out to be significantly different across the language groups despite the wiggle room that we allowed (i.e., small prior variance for the difference parameter). Thus, full approximate metric, but only partial approximate scalar invariance, may be claimed for the extraversion items. The same type of invariance (i.e., full approximate metric, partial approximate scalar) could be imposed for the other examined constructs (Table S2-S5), with exception of the concentration-stress test, for which full approximate metric and scalar invariance could be established (Table S1).

As a robustness check, we then reexamined the parameter differences with the subsampling procedure. Table 4 (column “Subsample comparisons”) shows the average parameter difference and the corresponding standard deviation across the subsample comparisons, as well as the percentage of subsample comparisons in which the parameter difference turned out to be significantly larger than zero. The results of the subsample comparisons of the other constructs are available as supplementary material (see supplementary material, Tables S1-S5, columns “Subsample comparisons”). Not surprisingly, parameter differences that had been discovered in the full sample comparison as significant also turned out to be significant in a large proportion of the subsample comparisons. For instance, the parameter difference of τ₂ was larger than zero in 100% of the subsample comparisons and that of τ₆ in 79% of the subsample comparisons (Table 4, column “Subsample comparisons”). More interesting, however, were the parameter differences that turned to be significant in a substantial proportion of the subsample comparisons (i.e., in 20%), even though the differences appeared to be non-significant in the full sample comparison. An exemplary case is the difference of τ₈ (row in gray, Table 4). Although this difference was not found to be significant in the full sample comparison, it proved to be significantly larger than zero in 52% of the subsample comparisons. Such masked differences were also found for the items of the other constructs (rows in gray, Tables S3-S5), even though the proportions of significant subsample comparisons were not as large (i.e., 20–25%) as was the case for τ₈ of the extraversion construct. Nevertheless, all these masked differences are certainly worth further examination.

Discussion

The aim of this study was to (re)examine selected GMA and personality tests used by the Swiss Armed Forces in psychological cadre assessment regarding model fit and measurement invariance across two major language groups in Switzerland (i.e., German and French speakers) by using BSEM.

The measurement model fit testing showed that the strictest BSEM model (i.e., the model with exact zero residual correlations) provided a bad fit to the data. Accordingly, if we had insisted on using the strict specifications that are applied in a conventional CFA with ML, the examination of measurement invariance of the constructs would have been over before it even began, as the central prerequisite of a well-fitting measurement model (Steenkamp & Baumgartner, 1998, p. 80) was not given. In contrast, more liberal BSEM models (i.e., models with approximate zero residual correlations) offered an adequate representation of the (co)variances in the data. By using liberal BSEM models, we could therefore obtain a well-fitting measurement model that could be used for further invariance testing. On the other hand, however, we had also to accept in turn that our measurement models and the underlying theories are at best approximations.

The subsequent measurement invariance testing then showed that at least partial approximate scalar invariance can be claimed for the items of the constructs. Interestingly, a large proportion of items that were noninvariant in the presently studied assessment cycle (i.e., 2019 cycle), were also found to be non-invariant in the previously studied assessment cycle (i.e., 2012 cycle; see Goldammer, 2019). For instance, all intercepts of the extraversion construct that turned out to be non-invariant in the present study (i.e., τ₂, τ₆, τ₇, τ₈, τ₉ in Table 4) were also found to function differently across the German- and French-speaking language groups in the 2012 assessment cycle (see Goldammer, 2019). This can be taken as further evidence for the robustness of the detected non-invariances, since they occurred in different samples and independently of the estimation method. However, this consistent finding also means that these non-invariances should be taken seriously.

Reflections on invariance testing and the assessment of the importance of non-invariances

Of course, the number of items detected to function differently across the groups depends strongly on how strict the desired invariance criterion is. Naturally, setting a stricter invariance criterion (e.g., exact invariance) most likely goes along with the finding that more items function differently across the groups. However, applying exact invariance as a criterion also entails the risk of finding many statistically significant yet potentially trivial non-invariances, such that not even a minimal level of invariance is reached that is needed to make meaningful comparisons across groups (e.g., Cieciuch, Davidov, Algesheimer, & Schmidt, 2018; Marsh et al., 2018). In light of the potential difficulties that may arise when using such a strict criterion of invariance, we therefore used BSEM as analytical framework that allows the application of a more lenient and potentially more realistic criterion of invariance (i.e., approximate invariance). More specifically, BSEM allows the researcher to anticipate a certain degree of trivial non-invariance, or to expect a certain degree of approximate invariance, by specifying a prior distribution with a mean of zero and a small variance for the difference between the parameters (e.g., loadings and/or intercepts). In this framework, parameter differences that are truly trivial therefore have credibility intervals that include zero. In contrast, if a difference between parameters turns out to be significant (i.e., the credibility interval does not include zero) despite the wiggle room that was allowed (i.e., set by the prior variance), the parameter difference may be considered to be non-trivial and thus of practical importance. This was also the criterion that we applied in our examination, when we assessed the practical importance of non-invariances. In other words, we considered any non-invariance that went beyond approximate invariance as worthy of further investigation (i.e., adjustments had to be made, or at least the items had to be examined more closely). We considered this criterion for the practical importance of a non-invariance as best suited for our high-stack questionnaire/test.

However, this is by far not the only possibility that researchers can use to determine the practical importance of a non-invariance. In the context of BSEM, for instance, Shi et al. (2019) proposed using the region of practical equivalence (ROPE) to determine whether a non-invariance is of practical relevance or not. For this type of non-invariance assessment, the researcher first defines a ROPE (preferably on standardized metric), such as ±.1, against which the credibility interval of the parameter difference can be compared (for details, see also Kruschke, 2014, p. 336–340). If the 95% credibility interval (preferably the highest density interval) falls completely within the ROPE, the researcher may conclude that the non-invariance is not of practical importance. On the other hand, if the 95% credibility interval lies completely outside the ROPE, the researcher may conclude that the parameter difference is of practical importance. However, the finding is considered as inconclusive if the ROPE and the credibility interval of the difference partially overlap (see Shi et al., 2019).⁸

These Bayesian-based approaches aside, there are also a lot of effect size measures available outside the Bayesian modeling and testing framework (for an overview, see Gunn, Grimm, & Edwards, 2020; Meade, 2010). In addition, rather than the item-level perspective that we used, one could use an effect size measure that determines the practical importance of a non-invariance by the extent to which it affects the overall scale score (see e.g., Stark, Chernyshenko, & Drasgow, 2004) or a measure that additionally takes into account the impact on the selection decision (see e.g., Stark et al., 2004). Accordingly, the most comprehensive picture of the practical importance of a non-invariance is likely to be obtained by considering several complementary indices (e.g., an item-level and scale-level index) (Stark et al., 2004, p. 498).

Among the multitude of available effect size measures of non-invariance, however, BSEM-based indices have a central advantage. Because with BSEM it is possible to obtain a theory-guided well-fitting measurement model with unbiased estimates, the effect size measures derived from the estimates are unbiased as well. In contrast, if a conventional CFA with ML is used, the derived effect size measures of non-invariance may be biased to the extent to which the CFA model is misspecified (see Nye & Drasgow, 2011, p. 977). In our understanding, this advantage of BSEM-based indices should therefore be clearly taken into account when researchers choose an effect size measure of non-invariance for their study of measurement invariance.

Different item functioning: Remedies and further testing options

Once a non-invariance has been detected, and judged as substantial, the question remains as to how to best deal with it. Clearly, more and more effective options are available if the causes of the non-invariance can be identified. If research is conducted in a multilingual context like ours, a good starting point for identifying the reasons for non-invariances is to check the translations of the items for imprecision. Indeed, an imprecise translation may have been the reason for the intercept difference in τ₂ (−.478) of the extraversion construct. Whereas the German item is, “Ich sprühe oft vor Energie” [I am often bursting with energy], the French item is, “je suis souvent plein d’énergie” [I am often full of energy]. Admittedly, this is only a slight nuance. However, the fact that the German item translation is stronger than the French translation may have ultimately led the German-speaking respondents to endorse it less often than the French-speaking respondents. If an imprecise translation has been identified as the most likely reason, as in our case, the adjustment is straightforward and entails finding a more suitable translation than the original one, or, if this is not possible, finding an expression that has the same meaning for both groups. In our case, this meant adjusting the German item wording to “Ich bin oft voller Energie” [I am often full of energy].

In addition to imprecise translations, however, non-invariances can also, and probably far more frequently, occur because the item triggers a different social norm or attitude in the language groups. For instance, behavior (e.g., outgoingness) in a social situation (e.g., at a party) may not only depend on the degree of extraversion but also on whether being reserved or being outgoing is considered as polite or as in accordance with the social norm in this circumstance. Under the premise that a concrete social norm can be identified as a reason for the non-invariance, the adjustment would entail replacing the item with one that does not trigger this norm but only indicates the intended construct. However, correctly attributing a non-invariance to a social norm is a challenging task and prone to speculations.

Eventually, the researcher may end up with a couple of non-invariant items for which no obvious reasons for the different functioning across the (language) groups can be found. However, this does not necessarily mean the end for any further tests and/or comparisons across groups. Even if several items are found to function differently, valid comparisons across groups may still be possible, as long as there are at least some invariant items in the questionnaire/test battery. For instance, Pokropek, Davidov, and Schmidt (2019) showed in their large simulation study that population latent means and path coefficients can be effectively recovered even if only some of the questionnaire items are approximately invariant. In such a case (i.e., partial approximate invariance) it seems most appropriate to use a BSEM-based CFA, as this analysis approach showed reasonable levels of coverage and precision even under unfavorable conditions (i.e., larger number of invariant items and larger prior variance for the difference parameters) and performed far better than other approaches examined (e.g., conventional multiple-group CFA with ML) (Pokropek et al., 2019).

Alternatively, if the comparability of individual questionnaire/test scores across groups is of primary interest, test equating may be applied (e.g., Kolen & Brennan, 2014). For instance, in our examination (i.e., a common item non-equivalent groups design), the set of invariant items could have served as internal anchor items, on which basis the different test versions (i.e., German and French versions) could have been equated. Mean, linear, or equipercentile equating are a few of the available methods that could have been used to put the scores of the language versions on the same scale (e.g., Bandalos, 2018, pp. 547–584; Kolen & Brennan, 2014, pp. 103–169). Readers looking for a concrete and illustrative example of how test equating may be applied in the context of linguistically different tests are referred to Elosua and López-Jáuregui (2008).

Limitations and future research directions

Some limitations should be noted. For instance, we used BSEM to better deal with models that fail the exact fit and/or invariance test, under the assumption that the approximate nature of the underlying measurement theory would be the primary reason for the model misfit. However, this perspective neglects the possibility that contextual factors of the survey administration may have also had a considerable influence on the participants’ response behavior. For one, there is the fact that the survey is used for selection purposes, which in turn may motivate certain cadre candidates to present themselves in a more favorable way (i.e., they fake some responses). Although there is an ongoing debate about the extent to which faking has adverse effects at all (for an overview, see Holden, 2008; Tracey, 2016), several studies have shown its distorting effects on the hypothesized factor structure (e.g., Ion & Iliescu, 2017; Lee, Joo, & Fyffe, 2019). For another, certain participants, presumably less motivated cadre candidates that had been implicitly coerced to take on a cadre position, may feel reluctant to go through the lengthy questionnaire and thus complete some parts or the whole questionnaire rather carelessly. This represents another potential response behavior with adverse effects on the factor structure (e.g., Goldammer, Annen, Stöckli, & Jonas, 2020). Future studies could therefore additionally include response style factors in their BSEM models. This would allow more clear differentiation between model misfit that is due to response styles and model misfit that may uniquely be attributed to the imprecision of the measurement model.

In addition, it can be argued that the criteria that we applied for assessing the practical importance of a non-invariance (i.e., anything that goes beyond approximate invariance) is only a heuristic in which the effective size of the non-invariance is not really taken into account. Obviously, in our approach to assessing the practical importance of a non-invariance, the size of the non-invariance/difference is not explicitly compared against a standardized metric, as is the case, for example, in the ROPE procedure outlined by Shi et al. (2019). However, it nevertheless entails a comparison, although a more implicit one, namely, in terms of prior variance that is specified for the difference between the parameters (e.g., loadings and intercepts). Naturally, in a strict BSEM model, in which only a small prior variance is specified for the difference between the parameters, it is likely that smaller non-invariances may turn out to be significant. But in more liberal BSEM models with more prior variance for the difference between the parameters, only substantial non-invariances may turn out to be significant. Accordingly, substantial differences can also be identified during the course of a sensitivity analysis, because substantial non-invariances turn out to be significant independent of the size of the prior variance that has been specified for the difference between the parameters (see supplementary material, Excel file “Invariance testing summary”). Nevertheless, future studies should examine to what extent the criterion that we used to judge a non-invariance as practically important (i.e., anything that goes beyond approximate invariance) leads to the same conclusions as other (BSEM-based) effect size measures. In this context, it would also be interesting to see which combination of different effect sizes measures (e.g., item- and test-level) provides the most comprehensive picture of the practical importance of different types of non-invariance (e.g., different number of non-invariant items, loading and/or intercept non-invariance, size of non-invariance).

Lastly, it might be argued that we focused (too) strongly on BSEM in our examination and thereby neglected other approaches that might be equally or even more suited. Undoubtedly, regularized structural equation models (RegSEM; Jacobucci, Grimm, & McArdle, 2016), BSEM-based alignment (Asparouhov & Muthén, 2014), and model implied instrumental variable two-stage least square estimation (MIIV-2SLS; Bollen, Gates, & Fisher, 2018) have their merits when dealing with models that fail the exact fit and/or invariance test. However, only few studies up to now have examined how these different approaches perform compared to each other (e.g., see Jacobucci & Grimm, 2018; Kim, Cao, Wang, & Nguyen, 2017, for a partial comparison between these approaches). Thus, at present, the researcher’s choice between these approaches is guided by relatively little information. Future studies should therefore examine under what types of misspecifications and/or non-invariances BSEM, RegSEM, BSEM-based alignment, and MIIV-2SLS perform best, so that applied researchers can make their choices on a more informed basis.

Conclusions

If GMA and personality constructs in a tailored psychological cadre assessment are meant to be valid in a cross-cultural context, two central prerequisites need to be given for the utilized measures. First, the scale items have to adequately reflect the underlying intended constructs. Second, the employed measures need to have the same psychometric properties in each cultural group. By examining these two important issues in the case of the multilingual psychological cadre assessment of the Swiss Armed Forces, this study helped to ensure the quality of this assessment and to identify possibilities for improvement. In a broader perspective, however, the study may have also provided guidance to researchers who have to deal with challenges similar to those that we faced (i.e., examining misfitting models for invariance in the context of unbalanced samples) and thus may help them to obtain robust findings despite these challenges.

Data availability statement

The data that support the findings of this study are available from the first author, upon reasonable request.

Disclosure statement

No potential conflict of interest was reported by the authors.

Supplementary material

Supplemental data for this article can be accessed on the publisher’s website.

Additional information

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Notes

1. In the concept development exercise, the candidates have 30 minutes to prepare a concept for a randomly assigned task (e.g., for a company outing, or a club’s anniversary). After the 30 minutes, the candidates hand in their concept on a single A4 sheet and present their solution in front of their peers and two psychologists.

2. The math test presents the candidates with six-digit numbers, and they have to decide whether the numbers can be divided by three and/or four or neither. At the beginning of this test, the candidates are given instructions on how to check whether a number is divisible by three or by four. Thus, this test should primarily measure numerological processing speed.

3. Initially, we wanted to test all constructs in a joint model, which would have entailed 1,698 freely estimated parameters in each language group, if approximate zero cross-loadings and residual correlations were specified. In such large model, however, a single iteration needed a processing time of approximately 30 seconds. We therefore decided to base our analyses on models that were computationally less demanding (i.e., we decided to run our tests for each construct separately), especially since we had planned to run several models with different priors (each with 50,000 iterations) to determine the best fitting model and additionally to run subsample comparisons with 100 repetitions for this final model.

4. The inverse Wishart (IW) settings (i.e., the settings of the degrees of freedom for the residual correlations) determined in the single group analyses with standardized factor variances and indicators were taken as proxy for the sample-specific IW settings when testing the unstandardized model estimates for measurement invariance across the language groups. This approach seemed justified, because the variances of the indicators tended to be close to 1 (see Muthén, 2013). However, the four indicators of the concentration-stress test were an exception. Here, a smaller prior variance had to be chosen than for the single-group analyses, since these unstandardized variances were remarkably different from 1 (i.e., on average only 0.0265).

5. For the appropriate comparison of structural parameters (e.g., latent means, or covariances) in the BSEM context, Muthén and Asparouhov (2013) proposed a two-step procedure: First, a model is run to identify non-invariant parameters. Second, a second model is run in which all parameters are constrained to be exactly equal except for those that turned out to be non-invariant (Muthén & Asparouhov, 2013). Because the comparison of structural parameters was not the focus of our study, we completed only the first of these two steps.

6. The subsampling approach also has its pitfalls. First, due to the reduced sample size, the overall power decreases and non-invariance may therefore remain undetected. Second, due to the reduced sample size, the influence of the prior on the posterior distribution of the difference parameter increases (see Muthén, Muthén, & Asparouhov, 2017, p. 385/415; Zyphur & Oswald, 2015, p. 398), which again can reduce the chance of detecting a substantial non-invariance. Thus, for drawing valid conclusions of the subsampling procedure, the size of the subsampled samples must be still large (powerful) enough to allow valid comparisons. Applying subsampling in the case of the German-speaking (n = 1,284) and French-speaking samples (n = 474) (i.e., using 474 randomly drawn German-speaking participants together with the 474 French-speaking participants for the invariance analysis) is justifiable, but it would hardly be justifiable if the Italian-speaking-group (n = 85) had been included as well (i.e., using 85 randomly drawn German-speaking and French-speaking participants together with the 85 Italian-speaking participants for the invariance analysis).

7. We chose to report the extraversion construct, because it best illustrated the point of the subsampling procedure, namely, to detect non-invariances that might be masked otherwise due to the unbalanced sample sizes.

8. In preliminary analyses, we also tried to examine the factorial invariance by using the procedure outlined by Shi et al. (2019). However, the specified models (i.e., configural models with language group-specific approximate zero residual correlations) almost never reached convergence, most likely because the models were too liberal (i.e., not identified).