Panel: Switching, Phasing-in and Improving: Multi-Armed Randomized Experiments with Multiple or Time-Contingent Components
(Methods and Tools of Analysis)

Friday, November 9, 2018: 8:30 AM-10:00 AM
Marriott Balcony B - Mezz Level (Marriott Wardman Park)

*Names in bold indicate Presenter

Panel Chairs:  Eric Hedberg, NORC at University of Chicago
Discussants:  Rob Olsen, Westat

The Design of Making Pre-K Count and High Fives: Two-Stage Random Assignment at Different Levels
Robin Jacob1, Marie-Andree Somers2, Shira Mattera2 and Pamela Morris3, (1)University of Michigan, (2)MDRC, (3)New York University

A Switching Replication with Multiple Treatments: An Example and Practical Guide
Kate Miller-Bains, Kylie L. Anglin, Emily Wiseman, Rebekah Berlin, Julie J. Cohen, Vivian C. Wong and Anandita Krishnamachari, University of Virginia

Insights from the Health Profession Opportunity Grant Program’s Three-Armed, Multi-Site Experiment for Program and Policy Learning and Evaluation Practice
Laura Peck1, Nicole Constance2 and Hilary Forster2, (1)Abt Associates, Inc., (2)U.S. Department of Health and Human Services

The proposed panel brings together researchers to discuss multi-armed randomized designs that involve phased program implementation. A fundamental question is: what sorts of research and policy questions can we answer using a phased-in multi-armed experiment? Such a design may reflect a specification of a program itself, if, for instance, the benefits of a second phase are expected to follow and perhaps be accelerated by completion of a first phase. In other applications the supplement added in phase-2 may be adaptive, reflecting what is learned during the first phase. Alternatively, a third arm may have to be added to allow an experiment to continue, as was the case with the well-known Tennessee Class Size Reduction experiment, where some normal-sized (control) classes were re-randomized to a “normal-sized class + an aide” condition to address stakeholders' concerns that not receiving the small class treatment is a certain disadvantage.         

Adding time as a factor introduces further complexity to a design that already has many facets to consider beyond a standard two-arm design. The participants in the proposed panel, drawing on work from multiple policy and content areas, will discuss their research with a view to phased introduction of program components. They will address the kinds of questions their designs provide answers to that cannot be addressed through more-standard multi-armed or even conventional two-armed designs, as well as the technical challenges that arise from using their variants of a multi-armed design.

The first paper describes a phased two-stage design used to examine the effects of aligned math programs across the pre-K and kindergarten years, with randomization conducted at different levels of the design.The second paper discusses a “multi-arm switching replication” approach to evaluating supports provided during simulated experiences in a teacher preparation program, with a view to the training situation that makes this design possible and to the trade-offs that come with using it.  The third paper examines impacts of a multi-faceted employment training program with individuals randomized to specific program components in addition to a standard program. The fourth paper proposes a multi-armed design alternative as a solution to the challenge of conducting a replication experiment involving a program that may have demonstrated adverse impact. A three-arm replication solution is intended to limit possible harm while allowing verification of the preliminary result.

The panel will inform attendees about technical, logistical, and applied aspects of time-contingent multi-armed designs with a view to answering policy-relevant questions.

See more of: Methods and Tools of Analysis
See more of: Panel