Fast Generalized Linear Models by Database Sampling and One-Step Polishing

Abstract

In this article, I show how to fit a generalized linear model to N observations on p variables stored in a relational database, using one sampling query and one aggregation query, as long as $N^{\frac{1}{2}+\delta }$ observations can be stored in memory, for some $\delta >0$. The resulting estimator is fully efficient and asymptotically equivalent to the maximum likelihood estimator, and so its variance can be estimated from the Fisher information in the usual way. A proof-of-concept implementation uses R with MonetDB and with SQLite, and could easily be adapted to other popular databases. I illustrate the approach with examples of taxi-trip data in New York City and factors related to car color in New Zealand. Supplementary materials for this article are available online.

Keywords:

• Big data
• Fisher scoring
• Maximum likelihood