Association for Public Policy Analysis & Management

Summer school has been identified as an effective mechanism to offset the slowdown or loss in academic performance that occurs during the 3-month summer break from schooling. The strength of the evidential basis warranting the impact of summer school is not without question however. Many authors have pointed to weaknesses in research design and implementation that serve to limit the inferences that can be drawn about the strength and generalizability of the summer school effect. In particular, the use of one group or nonequivalent group designs, full or partial noncompliance with the summer school offer, and the generalizability of estimates obtained from studies that utilize a regression discontinuity (RD) framework constrain our current understanding of when and for whom summer school is most beneficial.  As a result, the purpose of this paper is to demonstrate how an underutilized, but practical research design can be used to empirically address questions of conceptual, methodological, and policy interest. In the following, the application, analysis, and interpretation of a hybrid RD design is presented. The RD hybrid includes a randomized design component that increases power and strengthens inferences regarding program performance while affording the kind of generalization that has most value to practitioners and policy-makers.

 In RD designs, an intervention effect is revealed when a discontinuity in the regression function linking assignment (e.g., a pretest) and outcome variable scores is observed at the cutscore that defines the treatment and control conditions. Confidence in the estimate is enhanced when supplemental analyses confirm the model’s functional form and sensitivity tests of conceptually-relevant covariates reveal a balanced distribution on either side of the cutscore. Nonetheless, with increasing examination and use, it is now becoming clear that the traditional RD design has limitations that can reduce its utility. In addition to more onerous statistical modeling assumptions, the RD design tends to have lower statistical power and produce a less generalizable inference relative to a RCT. The constraint that limits the generality of the RD estimate follows from the unease associated with extrapolating the form of the regression function beyond the observed data. In the traditional RD design, it is not possible to know what the treatment group regression function would have been in the absence of treatment (i.e., the counterfactual), so inferences are judiciously limited to the narrow range of data surrounding the cut-score.

 The current demonstration employs a hybrid RD design (i.e., a combination of RD and RCT design components) to identify the effect of delivering supplemental summer instruction to a wider range of students. The intervention context is a 5-week summer program delivered to a sample of struggling readers and their more moderately at-risk peers who completed kindergarten in the prior academic year. Key implementation issues, including model specification, estimation, and sensitivity testing will be highlighted. The presentation will also consider how other RD design elements can be applied to strengthen inferences regarding program performance in contexts where it is not practical or ethical to withhold treatment from those in need.