Estimating Multilevel Models on Data Streams

L. Ippel*, M. C. Kaptein, Jeroen K. Vermunt

*Corresponding author for this work

Research output: Contribution to journalArticleAcademicpeer-review

Abstract

Social scientists are often faced with data that have a nested structure: pupils are nested within schools, employees are nested within companies, or repeated measurements are nested within individuals. Nested data are typically analyzed using multilevel models. However, when data sets are extremely large or when new data continuously augment the data set, estimating multilevel models can be challenging: the current algorithms used to fit multilevel models repeatedly revisit all data points and end up consuming much time and computer memory. This is especially troublesome when predictions are needed in real time and observations keep streaming in. We address this problem by introducing the Streaming Expectation Maximization Approximation (SEMA) algorithm for fitting multilevel models online (or row-by-row). In an extensive simulation study, we demonstrate the performance of SEMA compared to traditional methods of fitting multilevel models. Next, SEMA is used to analyze an empirical data stream. The accuracy of SEMA is competitive to current state-of-the-art methods while being orders of magnitude faster.

Original languageEnglish
Pages (from-to)41-64
Number of pages24
JournalPsychometrika
Volume84
Issue number1
DOIs
Publication statusPublished - Mar 2019

Keywords

  • Data streams
  • expectation maximization algorithm
  • multilevel models
  • machine (online) learning
  • SEMA
  • nested data
  • MAXIMUM-LIKELIHOOD
  • INCOMPLETE-DATA
  • EM ALGORITHM
  • ONLINE

Cite this