Association for Public Policy Analysis & Management

Increasingly, education researchers are examining experimental treatment effect variation across partially observed—or even fully latent—subgroups. For example, recent work has investigated whether the effect of Head Start varies by the type of care children would have experienced if they had not been offered enrollment in Head Start. Similarly, researchers might also want to determine whether a given educational intervention has a larger effect on students who would perform at the bottom of the achievement distribution in the absence of treatment. In both cases, we cannot directly observe students’ counterfactual behavior—for example, a child either received or did not receive a Head Start offer—but we can conceive of student membership in groups defined not only by their post-randomization action but the combination of this observed action and what their counterfactual behavior would have been.

We refer to groups defined by these potential behaviors unders assignement to treatment and control conditions as principal strata (Frangakis & Rubin, 2002). Under the principal stratification framework, a researcher defines endogenous (or program-related) subgroups (e.g., principal strata) based on sample members’ post-randomization choices, decisions, or experiences and under both the observed and unobserved (counterfactual) experimental conditions. While the principal stratification framework is helpful for articulating causal quantities of interest, estimation of treatment effects can be difficult in practice (e.g., Feller, Greif, Miratrix & Pillai, 2016).

In light of these estimation challenges, we examine the utility of relying on bounds for informing plausible ranges for principal causal effects of interest.  We articulate the estimation of bounds in the context of multi-site experimental data and discuss strategies for tightening bounds, including the use of covariates and the application of additional assumptions that are reasonable in the context of the specific interventions examined.  We utilize these bounding strategies to examine substantive questions related to two distinct experimental studies.  First, using data from an experimental study of Early College High Schools (ECHS) in North Carolina, we examine variation in the impact of ECHSs according to the quality of the high school students would otherwise attend.  Second, using data from a natural experiment of Small Schools of Choice in New York City, we examine whether high school graduation impacts were realized especially for those who were induced to be “on-track” for graduation at the end of the ninth grade. We show that in certain cases, meaningful substantive conclusions can be derived from bounds-based estimation of principal causal effects.