Large-Scale Generalized Linear Models for Longitudinal Data with Grouped Patterns of Unobserved Heterogeneity

ABSTRACT

This article provides methods for flexibly capturing unobservable heterogeneity from longitudinal data in the context of an exponential family of distributions. The group memberships of individual units are left unspecified, and their heterogeneity is influenced by group-specific unobservable factor structures. The model includes, as special cases, probit, logit, and Poisson regressions with interactive fixed effects along with unknown group membership. We discuss a computationally efficient estimation method and derive the corresponding asymptotic theory. Uniform consistency of the estimated group membership is established. To test heterogeneous regression coefficients within groups, we propose a Swamy-type test that allows for unobserved heterogeneity. We apply the proposed method to the study of market structure of the taxi industry in New York City. Our method unveils interesting and important insights from large-scale longitudinal data that consist of over 450 million data points.

KEYWORDS:

• Clustering
• Factor analysis
• Generalized linear models
• Interactive fixed effects
• Longitudinal data
• Unobserved heterogeneity

Supplementary Materials

All technical proofs are provided in the online supplementary document, along with Monte Carlo simulation results. The supplement also contains additional figures for the empirical analysis. The proposed methods are implementable through the R package PDMIF by Ando and Fayad (2022).

Acknowledgments

We thank the editor, Christian Hansen, an associate Editor and two anonymous referees for their constructive comments. We are also grateful for the comments and suggestions from the participants in the 14th International Conference on Computational and Financial Econometrics 2020 and the China Meeting of the Econometric Society 2021, and seminars at University of Queensland.