The Split-Ballot Multitrait-Multimethod Approach: Implementation and Problems

Abstract

Saris, Satorra, and Coenders (2004) proposed a new approach to estimate the quality of survey questions, combining the advantages of 2 existing approaches: the multitrait–multimethod (MTMM) and the split-ballot (SB) ones. Implemented in practice, this new approach led to frequent problems of nonconvergence and improper solutions. This article uses Monte Carlo simulations to understand why the SB-MTMM is working well in some cases but not in others. The number of SB groups is a crucial element: The 3-group design is performing better. However, the 2-group design can also perform well: The analyses suggest that the interaction between the absolute values of the correlations between the traits and the relative values of the different correlations between traits plays an important role.

Keywords:

• convergence
• Heywood cases
• Monte Carlo simulations
• quality of survey questions
• split-ballot multitrait–multimethod approach

Notes

For more information, please see http://www.europeansocialsurvey.org/.

Dutch Web panel based on probability sample. For more information, please see http://www.centerdata.nl/en/LISSpanel.

³An example of LISREL input to analyze SB-MTMM experiments is available online at http://www.upf.edu/survey/_pdf/note3.pdf.

⁴For more details about the traits and methods used in each experiment, as well as for the list of countries (or countries/languages groups) analyzed in each round, please see http://www.upf.edu/survey/_pdf/note4.pdf.

⁵To have population values corresponding to a PS, instead of doing an SB-MTMM, we combined the matrices for Groups 1 and 2, and filled in the part missing in both groups, trying to keep a similar pattern as for the rest of the matrix. Then, we estimated a one-group MTMM. The values chosen for Case 1 are based on these estimates.

⁶More details and the results can be found online at http://www.upf.edu/survey/_pdf/note6.pdf.

⁷An example of Mplus code is available at http://www.upf.edu/survey/_pdf/note7.pdf.

⁸The number of HC is not considered because we had no handy way of counting them. We would have needed to consider each replication separately because in the Mplus output of the Monte Carlo simulations we do not get indications about how many replications have improper estimates.

⁹The table with the results is available online at http://www.upf.edu/survey/_pdf/note9.pdf.

¹⁰As always, γ₁₄ = γ₂₄ = γ₃₄ = γ₄₅ = γ₅₅ = γ₆₅ = γ₇₆ = γ₈₆ = γ₉₆ = 1 and λ₁₁ = λ₂₂ = … = λ₉₉ = 1 and ϕ₁₁ = ϕ₂₂ = ϕ₃₃ = 1.

¹¹Please see http://www.upf.edu/survey/_pdf/note11.pdf.

*p < .05.

¹²We used estimates for ph21, ph31, and ph32 obtained by applying the SB-MTMM true score model in a multiple-group analysis with all the ESS countries for a given round included. These estimates can still be biased but we used them as an approximation we had of what the true correlations could be.