Association for Public Policy Analysis & Management

Panel: Switching, Phasing-in and Improving: Multi-Armed Randomized Experiments with Multiple or Time-Contingent Components

(Methods and Tools of Analysis)

The proposed panel brings together researchers to discuss multi-armed randomized designs that involve phased program implementation. A fundamental question is: what sorts of research and policy questions can we answer using a phased-in multi-armed experiment? Such a design may reflect a specification of a program itself, if, for instance, the benefits of a second phase are expected to follow and perhaps be accelerated by completion of a first phase. In other applications the supplement added in phase-2 may be adaptive, reflecting what is learned during the first phase. Alternatively, a third arm may have to be added to allow an experiment to continue, as was the case with the well-known Tennessee Class Size Reduction experiment, where some normal-sized (control) classes were re-randomized to a “normal-sized class + an aide” condition to address stakeholders' concerns that not receiving the small class treatment is a certain disadvantage.         

Adding time as a factor introduces further complexity to a design that already has many facets to consider beyond a standard two-arm design. The participants in the proposed panel, drawing on work from multiple policy and content areas, will discuss their research with a view to phased introduction of program components. They will address the kinds of questions their designs provide answers to that cannot be addressed through more-standard multi-armed or even conventional two-armed designs, as well as the technical challenges that arise from using their variants of a multi-armed design.

The first paper describes a phased two-stage design used to examine the effects of aligned math programs across the pre-K and kindergarten years, with randomization conducted at different levels of the design.The second paper discusses a “multi-arm switching replication” approach to evaluating supports provided during simulated experiences in a teacher preparation program, with a view to the training situation that makes this design possible and to the trade-offs that come with using it.  The third paper examines impacts of a multi-faceted employment training program with individuals randomized to specific program components in addition to a standard program. The fourth paper proposes a multi-armed design alternative as a solution to the challenge of conducting a replication experiment involving a program that may have demonstrated adverse impact. A three-arm replication solution is intended to limit possible harm while allowing verification of the preliminary result.

The panel will inform attendees about technical, logistical, and applied aspects of time-contingent multi-armed designs with a view to answering policy-relevant questions.