In many evaluations in the social and medical sciences individuals are randomly assigned to a treatment arm or a control arm of the experiment. Often, after treatment assignment is determined, individuals within one or both experimental arms are grouped together (e.g., within classrooms/schools, through shared case-managers, in group therapy sessions, through shared doctors, etc.) to receive services. Consequently, there may be within group correlations in outcomes resulting from at least three sources: (1) the process that sorts individuals into groups, (2) service provider effects, and/or (3) peer effects. When estimating the standard error of the impact estimate, it may be necessary to account for within-group correlations in outcomes (also described as “lack of independence of observations”). Past research has focused on whether to consider service providers as “fixed” or “random” effects in this situation. This work is agnostic on that issue, but demonstrates the challenge that the sorting process and resulting shared experience of individuals creates for estimating the standard error of the impact estimate.

This work builds off of previous research on the appropriate interpretation and analysis of data in individually randomized trials. The talk will cover:

1. A review of the implications of service provider selection and effects when defining “the causal” effects in individually randomized experiments.  The review will describe how individual randomization can make it difficult or impossible to disentangle the effect due to the intervention from the effect due to the service providers. While not the central theme of this work, clarity on this issue is fundamental.
2. The three potential sources of lack of independence of observations in individually randomized experiments: (1) non-random sorting into groups, (2) provider effects, and (3) peer effects. Importantly, these sources cannot be disentangled without modeling assumptions, but may need to be accounted for in analyses.
3. The fixed and random effects approaches to accounting for lack of independence of observations, which have been described in detail in past literature.  This includes discussion of the intended inferences of each approach (this is not intended to be new, but rather it is necessary background information).
4. Presentation of analytical results and a related set of simulations that demonstrate and elucidate when fixed effects are appropriate, when random effects are appropriate, and when neither are appropriate. In addition, the simulations provide control over the process that sorts individuals into groups – permitting a demonstration of how this sorting can bias the estimates of the standard error of the impact estimates under both the fixed and random effects paradigms.
5. Suggestions on how to proceed in real world experiments.