Association for Public Policy Analysis & Management

Moderation analysis can be used to answer the questions “work for whom, and under what conditions”. The potential moderators of the intervention in cluster randomized trials (CRTs) include pretest, ethnicity, school climate, or the fidelity of implementation, which could be at different levels and have different distributions (e.g., binary, continuous). Like studies that focus on detecting a main effect, a critical consideration in designing studies to detect moderation effects is the statistical power with which the moderation effect can be detected if they exist. For moderator relationships in experimental studies, Bloom (2005) and Spybrook (2012) have presented procedures for conducting power analysis for binary moderators in two- to four-level cluster randomized experiments, but have not extended those procedures to include continuous moderator variables. Furthermore, no any computational tools to facilitate use of these techniques by researchers have been developed. Most recently, Mathieu, Aguinis, Culpepper, & Chen (2012) conducted a comprehensive Monte Carlo simulation to estimate the statistical power to detect cross-level interaction effects. However, Mathieu et al (2012) only studied two-level analysis without including covariates, and did not provided closed form formulas to estimate the statistical power, minimum detectable effect size, or minimum required sample size to detect meaningful effects.

In this study, we developed the statistical formulations for calculating statistical power, minimum detectable effect size and its confidence interval, and sample size to detect continuous moderator effects in CRTs. The moderator and other covariates could be at level 1 or level 2. We will also embody those formulations in the enhanced version of PowerUp! (Dong & Maynard, 2013) readily available to researchers for use in conducting continuous power analysis when planning CRTs focused on these relationships. We will make recommendations for the design of CRTs to detect the continuous moderator effects.