The Design of Making Pre-K Count and High Fives: Two-Stage Random Assignment at Different Levels
*Names in bold indicate Presenter
The Making Pre-K Count and High 5s studies are a recent application of a phased two-stage design used to examine the effects of aligned math programs across the pre-K and kindergarten years. The studies were designed to rigorously test whether providing aligned, high-quality math instruction during pre-K and kindergarten could provide a critical boost that would lead to long-term achievement gains across a variety of domains. In Making Pre-K Count, pre-K centers were randomly assigned either to receive an evidence-based early math curriculum (Building Blocks) and associated professional development or to a pre-K-as-usual control condition. In High 5s, kindergarten students who had been in Making Pre-K Count program classrooms in pre-K were then individually randomly assigned within schools to small-group supplemental math clubs that were designed to sustain the gains from the pre-K program, or to a business-as-usual kindergarten experience. Uniquely, the level of random assignment differs across stages: in the first year, a cluster-level random assignment design was used to randomly assign full pre-k schools; in the second year, individual-level random assignment was used to randomly assign students from the first-stage treatment group.
This design created three groups at the end of kindergarten: 1) children with 2 years of math enrichment (MPC in pre-k and High 5s in kindergarten); 2) children with 1 year of math enrichment (MPC in pre-k only); and 3) children with no math enrichment (pre-k and kindergarten as usual control group). This two-stage design allowed for three questions: whether the pre-k program or the kindergarten program worked, and also how the alignment of the two worked above and beyond any one.
The design raised novel logistical and analytic challenges. The presentation will highlight some of these challenges, including issues of consent and power, as well as lessons learned about how they might be overcome. The presentation will present a set of sensitivity analyses that were used to examine how these issues influenced program impact estimates and how different analytic options could help address some of the design issues. Despite these challenges, this design balanced multiple considerations of power, the number of research questions that could be answered, and efficiency of resources.