Experiments are held as the “gold standard” in the social and educational sciences since, as a result of random assignment to treatment conditions, they allow for unbiased estimation of the causal impact of a treatment or intervention. The policy problem with interpreting the results of these experiments is that the sample selected is rarely randomly selected from a well defined inference population. Recently this issue of generalizability has received renewed interest and methods for improving estimates of and sample selection for estimating the population average treatment effect have been proposed. When treatment effects are heterogeneous, however, the population average treatment effect may not be the only estimand of interest. For example, policy makers may also be interested in the understanding the degree of heterogeneity of these effects, or in understanding for whom a treatment works best. In this paper, we propose that these three estimands – the average treatment impact, the variability in treatment impacts, and moderators of treatment impacts – lead to different sample selection goals. We discuss the implications of these estimands and their associated estimators on the choice of a sample selection strategy. We situate this work in the analysis of a large-scale cluster-randomized evaluation of a math curriculum.