Association for Public Policy Analysis & Management

Cost-effectiveness analysis (CEA) is a method of comparing the cost and effectiveness of alternative programs, policies, and practices with similar goals (Levin & Belfield, 2015). It has been used most widely in assessments of health care interventions and medical practices. Recently, with the increasing investment in assessing the effectiveness of educational programs using rigorous methods that often include randomizing students, classrooms, or schools to treatment or control condition, there is growing interest in synthesizing and comparing the results across educational interventions. When comparing findings across interventions, it is important to consider not only the relative effectiveness in changing the focal outcome, but also to take account of the relative cost of achieving a unit change. It is increasingly common for impact evaluations of educational interventions to also include rigorous cost analysis components to evaluate the cost-effectiveness of the treatment and to support subsequent cross-study comparisons of the cost-effectiveness of alternative strategies for achieving target educational outcomes (Hollands et al., 2016).

As with conventional impact evaluations, it is important to conduct a priori power computation when designing randomized trials aimed at estimating the cost-effectiveness of the treatment (commonly referred to as randomized cost-effectiveness trials or RCETs). The objective of this analysis is to ensure that the sample design offers a “good enough” chance (e.g., p ≥ 0.80) to detect the cost-effectiveness of the treatment.

Methods of conducting power analysis for RCETs in the health and medical sciences have been developed (e.g., Willan & Briggs, 2006). Recent work in health economics (Manju, Candel, & Berger, 2014, 2015) used two-level (e.g., students nested within classrooms) hierarchical linear models (HLMs) to address the potential nesting effects and provides formulas to calculate power for two-level RCETs. However, these approaches cannot be adopted easily by educational researchers for three reasons: (1) their methods assume individual level cost information is available, while in education such information is usually missing (Levin & Belfield, 2015); (2) their models do not account for the effects of covariates, which are commonly available and can impact quite considerably the required sample sizes in education studies (Hedges & Hedberg, 2007); and (3) educational interventions usually have more complicated nesting structure (e.g., students nested with classroom, and classrooms nested within schools).

This study contributes to CEA and power analysis literature by extending prior work to accommodate three-level models, various assumptions about covariates, and alternative sample designs and analytic modeling assumptions. Specifically, we develop new formulas for estimating the statistical power of two- and three-level cluster RCETs with four alternative designs varying by the total number of levels, the level of treatment assignment, and the level of the available cost information. In general, the power computation takes account of sample sizes (e.g., the number of students, classrooms, and schools), effect size, covariates effects, nesting effects (i.e., intra-class correlations for both cost and effectiveness data), and the correlations between cost and effectiveness at each level. We demonstrate how to design cluster RCETs with adequate power using the framework of PowerUp! (Dong & Maynard, 2013).