Association for Public Policy Analysis & Management

In education and social welfare, policy makers are often interested in understanding if a program or intervention is worth investing in and implementing at scale. It is therefore incumbent upon researchers conducting evaluations of these interventions to not only plan for the ability to detect causality, but also to determine if the impact found in the study might generalize to a population of policy relevance. The simplest statistical method for making these generalizations is to first randomly select the sample of sites from a well-defined inference population, and second to randomly assign units within this sample to receive the intervention. This dual randomization process is incredibly rare in practice however, with best practice typically including random assignment within a convenience sample. In response to this reality, recent statistical work has focused on the development of alternative methods for making generalizations, including methods for site selection, which are the focus of this paper. These methods require researchers to first identify an inference population and then specify a large list of covariates that can be used for stratification; these continuous and categorical variables are reduced to a single dimension for stratification via use of propensity scores or cluster analysis methodologies. Sites are then selected from each stratum with a goal of estimating an (conditionally) unbiased average treatment effect for the population.

The focus of this paper is on design and planning considerations for these methods, particularly in relation to tradeoffs between the MDES and power under different types of stratification designs and sample allocations. We focus here on three design parameters: the number of strata (k), the degree of homogeneity in the covariates within these strata (H), and vectors of weights indicating the allocation of sites to the strata in the sample (w_sj = (w_s1, …, w_sk)) and population (w_pj = (w_p1,…,w_pk)).  While previous work has focused on the optimal conditions – where k is selected so that H is large, and w_sj = w_pj for all strata – in real applications these are not always possible. In the paper we provide analytic results that can be used in conjunction with standard power analysis methodologies – e.g., a design effect based on stratum allocations (Σw_pj²/w_sj) – as well as examples.