The dubious benefits of multi‐level modeling

Abstract

This paper presents an argument against the wider adoption of complex forms of data analysis, using multi‐level modeling (MLM) as an extended case study. MLM was devised to overcome some deficiencies in existing datasets, such as the bias caused by clustering. The paper suggests that MLM has an unclear theoretical and empirical basis, has not led to important practical research results, is largely unnecessary due to the availability of a range of alternatives, and is therefore overly complex, making research reports harder to read and understand for a general audience for no apparent analytic gain. Above all, the paper shows via examples that MLM has made so little difference in practice that it is worth us also considering its analytic costs. These costs include the promotion of an educational form of ‘asterix economics’, the creation of unworkable premises such as denying the existence of population data, and the tension between the contradictory need for both a large and small number of cases at each sub‐level. The paper concludes by outlining a substantial number of alternative methods of analysis which can have the same effect as MLM in examining structures in the data or overcoming the problems caused by clustering. Many of these alternatives are easier to use and understand, do not require specialist software, and avoid the problems—such as having to ignore cases with missing variables—created by the use of MLM.

Acknowledgement

I would like to thank Patrick White for frequently sending me links to interesting articles that I might otherwise have missed.

Notes

1. For example, the simple denary fraction 1/10 can be represented in eight binary digits as 0.00011001. This is the most accurate representation of 1/10 in eight bits, but would convert back to decimal as only 0.9765625. Of course, most computers and calculators now use many more than eight bits to store each number (or actually the mantissa in standard floating point format), but the same principle applies. Using more bits increases the accuracy of converting from decimal to binary, but such representational errors will always remain. These errors will propagate through calculations, either increasing or decreasing relative to the intended results.