Association for Public Policy Analysis & Management

Multi-arm designs have the advantage of providing greater flexibility to evaluate multi-component or sequenced interventions. A two-stage approach to random assignment can be used to create a multi-arm design. In the first stage, participants are randomly assigned to an intervention; in the second stage, the treatment group is re-randomized into another intervention. This design creates two program groups –one group that received both interventions, and one group that received the first intervention – and a control group, which allows for the ability to test the combined and relative effect of the two interventions. Such designs are particularly intriguing for examining the effect of interventions that are sequenced over time as children move from preschool to kindergarten.

The Making Pre-K Count and High 5s studies are a recent application of a phased two-stage design used to examine the effects of aligned math programs across the pre-K and kindergarten years. The studies were designed to rigorously test whether providing aligned, high-quality math instruction during pre-K and kindergarten could provide a critical boost that would lead to long-term achievement gains across a variety of domains. In Making Pre-K Count, pre-K centers were randomly assigned either to receive an evidence-based early math curriculum (Building Blocks) and associated professional development or to a pre-K-as-usual control condition. In High 5s, kindergarten students who had been in Making Pre-K Count program classrooms in pre-K were then individually randomly assigned within schools to small-group supplemental math clubs that were designed to sustain the gains from the pre-K program, or to a business-as-usual kindergarten experience. Uniquely, the level of random assignment differs across stages: in the first year, a cluster-level random assignment design was used to randomly assign full pre-k schools; in the second year, individual-level random assignment was used to randomly assign students from the first-stage treatment group.

This design created three groups at the end of kindergarten: 1) children with 2 years of math enrichment (MPC in pre-k and High 5s in kindergarten); 2) children with 1 year of math enrichment (MPC in pre-k only); and 3) children with no math enrichment (pre-k and kindergarten as usual control group). This two-stage design allowed for three questions: whether the pre-k program or the kindergarten program worked, and also how the alignment of the two worked above and beyond any one.

The design raised novel logistical and analytic challenges. The presentation will highlight some of these challenges, including issues of consent and power, as well as lessons learned about how they might be overcome. The presentation will present a set of sensitivity analyses that were used to examine how these issues influenced program impact estimates and how different analytic options could help address some of the design issues. Despite these challenges, this design balanced multiple considerations of power, the number of research questions that could be answered, and efficiency of resources.