Association for Public Policy Analysis & Management

A ubiquitous---if often implicit---assumption in the analysis of multi-site trials is that the distribution of site-level average treatment effects is Normal. While this is a convenient modeling assumption, it is rarely justified in practice, and, if incorrect, can yield poor estimates for quantities such as the proportion of sites with a negative treatment effect. In this paper, we show that the problem is effectively one of density estimation with measurement error, and, by utilizing the Central Limit Theorem within each site, we can precisely characterize this measurement error. We then discuss different approaches, such as Normal mixture models and Dirichlet Processes, for estimating the underlying treatment effect distribution. Finally, we extend this approach to investigate the relationship between site-level covariates and treatment effects. Our main motivating example is a large randomized evaluation of Early College High Schools in North Carolina. We find that standard results relying on Normality mask interesting and important results from this study.