Design-based analysis of experiments embedded in probability samples

This chapter presents a general design-based framework for the design and analysis of experiments embedded in probability samples. It discusses design considerations for experiments embedded in probability samples, and gives a design-based inference approach for single-factor experiments where sampling units are also the experimental units. The chapter develops a theoretical framework to test hypotheses about differences between finite population parameter estimates observed under different survey implementations or treatments, based on a field experiment embedded in a probability sample. Explaining systematic differences between a target variable observed under different treatments or survey implementations requires a measurement error model. The chapter considers special cases where the proposed procedure coincides with the more familiar model-based analysis such as ANOVA F-tests or two sample i-tests. It also presents an experiment with different data collection modes in the Dutch Crime Victimization Survey as an illustrative example.