Association for Public Policy Analysis & Management

In 2015, the Administration for Children and Families (ACF) contracted with Mathematica Policy Research and its partner, Public Strategies, to conduct the Strengthening Relationship Education and Marriage Services (STREAMS) evaluation. STREAMS is a five-site, random assignment evaluation of Healthy Marriage and Relationship Education (HMRE) programs. At one of the STREAMS sites, Mathematica is examining the effectiveness of different types of text messages to improve program attendance at HMRE workshops for adult couples. This study leverages a Bayesian adaptive design to adjust the random assignment probabilities for assigning couples to different text messaging interventions over time, which increases statistical power to detect significant treatment effects. This approach also allows us to test a greater number of text messaging interventions over the lifetime of the study, replacing ineffective treatments with new ones.

Prior studies of HMRE programs have shown that programs can face challenges in motivating enrolled couples to regularly attend voluntary workshop sessions (Dion et al., 2010; Miller et al., 2012; Zaveri and Baumgartner, 2016). This study builds on the large and growing behavioral science literature from outside the field of HMRE programming, which suggests that text messages informed by behavioral science can improve program participation and attendance (see Hasvold and Wootton, 2011) and that reminders can increase the rate at which people complete their intended actions (Bergman, 2015; Castleman and Page, 2015; Mullainathan and Shafir, 2013). The evaluation team will analyze program attendance data on 1,500 couples participating in relationship education workshops across five Florida counties to understand whether text messages informed by behavioral science can serve as a practical, relatively low-cost strategy for improving couples’ attendance at HMRE workshops. The evaluation will also examine the relative effectiveness of the specific content of different messages.

The evaluation’s random assignment approach uses a Bayesian adaptive design that enables the evaluation team to adjust the random assignment probabilities for assigning couples to research groups over time (Finucane et al., 2018). Initially, the evaluation team randomly assigned couples in roughly equal numbers to one of four research groups. Those in the treatment groups received one of the three behaviorally informed text messaging interventions, and those in the control group did not receive a text messaging intervention. After each cohort of couples completed their HMRE workshop, the evaluation team analyzed the attendance rates for each research group and adjusted the random assignment probabilities to allocate more couples to the interventions that appeared most promising. When the data clearly established that some interventions were no more effective than the control, the evaluation team stopped assigning couples to those interventions and replaced them with new interventions to be tested. Thus, the Bayesian adaptive design enabled the evaluation team to use a data-driven random assignment strategy to test a larger number of interventions with greater statistical power than would be possible under a traditional random assignment approach. This presentation will discuss the study design, interim results, and practical considerations for conducting the study. Final results from this study will be available in early 2020.