Association for Public Policy Analysis & Management

The fidelity of program implementation and the compliance of participants in social interventions and clinical trials have received increasing attention from practitioners, developmental and policy scientists, administrators, and policymakers. Most experimental interventions have suffered from complications due to flaws in program implementation and participants' noncompliance. One major empirical hurdle in examining how fidelity and compliance are related to program impact is the persistent issue of selection bias. Participants' characteristics may affect both implementation fidelity and outcomes of interest. Similarly, participants assigned to the intervention group may choose whether to comply with the assignment to participate, or to what extent to participate. Moreover, in randomized interventions, individuals in the control group are not exposed to intervention, and thus the levels of fidelity and compliance that would have resulted if assigned to the intervention condition cannot be directly observed.

This study capitalizes on the recently collected data in the Cultivating Awareness and Resilience in Education (CARE) project to investigate whether different levels of fidelity and compliance have different impact on teachers' emotional well-being and efficacy as well as the quality of classroom interactions. Overall, 224 general education teachers of grades K-5 from 36 public elementary schools in a high poverty region of New York City participated during 2012-2014. As the largest randomized controlled trial (RCT) of a mindfulness-based intervention for teachers to date, teachers randomized within schools to the CARE professional development program were presented with a structured set of mindful awareness practices as well as didactic and experiential practices regarding emotion awareness and emotion regulation training and compassion training.

Fidelity of implementation, assessed by trained fidelity coders, was measured by adherence and quality to capture the completion of components, using the CARE Daily Session Rating Forms; and how well the participant learning objectives were met, using the CARE Facilitator Rating Form, a modified version of the Iowa Strengthening Families Program Facilitator Delivery Ratings. Program compliance was measured by participants' attendance using a daily attendance log. Teachers' emotional well-being and efficacy were assessed by an online battery of self-reported measures on personal distress (e.g., depression, anxiety, perceived stress, and sleep), burnout/time pressure, efficacy, and mindfulness. The quality of classroom interactions was assessed by trained, independent observers using the Classroom Assessment Scoring System (CLASS), including emotional support, classroom organization, and instructional support. To address the issue of selection bias, this study uses both principal score matching and inverse probability weighting to identify and compare teachers in the control group who had similar characteristics to teachers in the intervention group who experienced different levels of fidelity and compliance. A mediating analysis is further conducted to examine whether variation in CARE implementation fidelity has indirect effects on classroom quality through changes in teachers' emotional well-being and efficacy.

Findings from this project will help the field understand the impact of different levels of fidelity and compliance on developmental outcomes among participants, identify potential predictors of the variation in fidelity and participant compliance, and strengthen the design, implementation, evaluation and scale-up of effective social and clinical interventions.