Association for Public Policy Analysis & Management

Multi-arm randomized control trials allow researchers to answer questions about the comparative effectiveness of more than one program or program component. However, executing multi-arm RCTs in real-world contexts is complicated, and the internal validity of these studies can be threatened by common occurrences such as participant attrition, small sample sizes, and treatment contamination. In one underutilized variant of the RCT known as a switching replication (Edmonds & Kennedy, 2017; Shadish, Cook, & Campbell, 2002), the researcher randomly assigns participants to different treatment arms and switches the treatment conditions at a later time point. In a traditional switching replication, the researcher collects outcome data on all participants at three time points: a pretest prior to treatment administration, a midtest after the first treatment administration, and a posttest after a second treatment administration. In the treatment repetition, the previous treatment group serves as the control and the original control group receives treatment. In a multi-arm switching replication framework, the treatment groups rotate through multiple treatment conditions (see Table 1).

The switching replication design can counter common concerns in experimental studies. First, because no one is denied treatment, the switching replication can help improve the political feasibility of randomization and reduce the threat of compensatory rivalry or demoralization. Secondly, by leveraging multiple independent administrations of the treatment, the switching replication has the potential to improve external validity by demonstrating the consistency of treatment effects. In instances where the treatment is not expected to have a lasting effect, the researcher can improve statistical power by collapsing treatment and control groups across the treatment administrations. Alternatively, when analyzing the treatment administrations independently, the design allows researchers to evaluate both the short- and long-term effects of a given set or combination of interventions.

However, the design possesses its own set of considerations for implementation and analysis, some to which we allude in the preceding sections. The switching replication requires multiple waves of data collection and treatment implementation. This means that it is best suited for situations in which interventions can be repeated at regular intervals, with out and ideally in a context with low risk of participant attrition after the first treatment. Furthermore, because the data in a multi-arm switching replication can be analyzed in multiple ways to address more than one research question, the researcher must properly account for multiple comparisons and consider power differences across questions.

In this paper, we discuss the practical and analytic considerations in the use of a multi-arm switching replication to evaluate supports provided during simulated learning experiences in a teacher preparation program. We will discuss the challenges we encounter as well as the potential advantages and trade-offs of the multi-arm switching replication in practice, along with some preliminary results. We also examine the various assumptions underpinning the design and analytic considerations one must make when implementing this design. We hope that it will serve as a practical guide to planning and implementing this underutilized design in future social science research.