Switching, Phasing-in and Improving: Multi-Armed Randomized Experiments with Multiple or Time-Contingent Components
(Methods and Tools of Analysis)
*Names in bold indicate Presenter
Adding time as a factor introduces further complexity to a design that already has many facets to consider beyond a standard two-arm design. The participants in the proposed panel, drawing on work from multiple policy and content areas, will discuss their research with a view to phased introduction of program components. They will address the kinds of questions their designs provide answers to that cannot be addressed through more-standard multi-armed or even conventional two-armed designs, as well as the technical challenges that arise from using their variants of a multi-armed design.
The first paper describes a phased two-stage design used to examine the effects of aligned math programs across the pre-K and kindergarten years, with randomization conducted at different levels of the design.The second paper discusses a “multi-arm switching replication” approach to evaluating supports provided during simulated experiences in a teacher preparation program, with a view to the training situation that makes this design possible and to the trade-offs that come with using it. The third paper examines impacts of a multi-faceted employment training program with individuals randomized to specific program components in addition to a standard program. The fourth paper proposes a multi-armed design alternative as a solution to the challenge of conducting a replication experiment involving a program that may have demonstrated adverse impact. A three-arm replication solution is intended to limit possible harm while allowing verification of the preliminary result.
The panel will inform attendees about technical, logistical, and applied aspects of time-contingent multi-armed designs with a view to answering policy-relevant questions.