Dealing With Data Streams

Lianne Ippel*, Maurits Kaptein, Jeroen Vermunt

*Corresponding author for this work

Research output: Contribution to journalArticleAcademicpeer-review


Novel technological advances allow distributed and automatic measurement of human behavior. While these technologies provide exciting new research opportunities, they also provide challenges: datasets collected using new technologies grow increasingly large, and in many applications the collected data are continuously augmented. These data streams make the standard computation of well-known estimators inefficient as the computation has to be repeated each time a new data point enters. In this tutorial paper, we detail online learning, an analysis method that facilitates the efficient analysis of Big Data and continuous data streams. We illustrate how common analysis methods can be adapted for use with Big Data using an online, or "row-by-row," processing approach. We present several simple ( and exact) examples of the online estimation and discuss Stochastic Gradient Descent as a general (approximate) approach to estimate more complex models. We end this article with a discussion of the methodological challenges that remain.
Original languageEnglish
Pages (from-to)124-138
Number of pages15
JournalMethodology-European Journal of Research Methods for the Behavioral and Social Sciences
Publication statusPublished - 2016
Externally publishedYes


Dive into the research topics of 'Dealing With Data Streams'. Together they form a unique fingerprint.

Cite this