Association for Public Policy Analysis & Management

Treatment effect heterogeneity is a critical component in understanding the results of large-scale randomized trials. A first step in an analysis of such variation might be to test for the presence of variation overall (i.e., to test for idiosyncratic variation not fully modeled by covariates) before tying the variation to specific covariates (to ideally obtain a model of systematic, or explainable, variation). The question is then how to conduct such an initial first-step omnibus test in a maximally powerful way. This talk first compares two classic methods for detecting such variation, and then extends these methods to take advantage of site level covariates that might partially predict such variation. The two fundamental methods examined are a test built on ANOVA-style variance calculations borrowed from meta-analysis and a likelihood ratio test for a variance component in a multilevel model. Neither of these tests use covariate information to model heterogeneity. Alternatively, one could test for the presence of an interaction term between a given site-level covariate and treatment in a linear model, but if much of the variation is unrelated to such a covariate, this test also may not be optimal. We propose to therefore use a hybrid test that tests for both systematic and idiosyncratic variation simultaneously using an adjusted likelihood ratio test. Overall, we examine two primary methodological research questions: (1) What methods are most powerful for detecting cross site variation, and why? and (2) How can one best exploit a covariates modestly predictive of variation to improve the power of an overall test?