Unbalanced 3-Group Split-Ballot Multitrait–Multimethod Design?

Abstract

A common way of estimating measurement quality is the split-ballot multitrait–multimethod (SB-MTMM) approach. However, this approach leads often to non-convergence or improper solutions when using a 2-group design, whereas the 3-group design performs better. Nevertheless, the 3-group design is rarely implemented because it makes it complicated for applied researchers to use the data. Therefore, we propose to draw groups of unequal sample sizes: two larger groups and one third group as small as possible. Using Monte Carlo simulations and real data analyses, we investigate how well such a design works and which size is needed for the third group. Our results suggest that a 3-group SB-MTMM design with smaller size for the third group (reducing till 5–10%) leads to similar levels of accuracy and no large changes in the model or quality estimates.

Keywords:

• 3-group design
• convergence
• European Social Survey
• measurement quality
• Monte Carlo simulations
• proper solutions
• split-ballot multitrait–multimethod (SB-MTMM) approach

Acknowledgments

We are very thankful to Willem Saris and Albert Satorra for all the discussions we had over the years, during which the main idea of this paper was developed, and the Core Scientific Team of the European Social Survey ERIC for its sustained support of this research.

Notes

¹ In order to test the robustness of this result, similar simulations were also done for all other sets of values and conditions which were qualified as poor or quite poor in Revilla and Saris (2013, Table 8, p.41, column “CCL”) and for some conditions where no problems were encountered. The same pattern is found in all cases: even with only 2% of the observations in the third group, almost all models converge and the bias and MSE are low on average. However, the standard deviations are usually larger with 2%, while already having 5% leads to quite similar standard deviations than having balanced groups.

² This is different from what is done in the simulation because we were not able to collect real data for different settings. The real data were collected using a balanced design. We then use these data to subsample smaller third groups, but we cannot reattribute the observations lost in group 3 to groups 1 and 2. This means that the total sample size differs in this case across the different models tested, which affects the testing of the model. While the way of analyzing used in the simulations is more appropriate, because of the limits of the available data, in the real data analyses, we had to do it in another (less optimal) way.

³ https://www.europeansocialsurvey.org/download.html?file=ESS8MTMMe02&y=2016.