Information Benefits and Statistical Challenges of Complex Multi-Armed Trials: Innovative Designs for Nuanced Questions
(Methods and Tools of Analysis)
*Names in bold indicate Presenter
Paper 1 demonstrates an approach that takes advantage of a multi-armed design to assess impacts of program components that are voluntary or based on encouragement. It focuses on estimation of the total combined effect, impact of the main treatment component, and the added value of the optional component; Paper 2 describes a multi-armed design that accommodates both population and volunteer experiments allowing comparisons of results and tradeoffs in external validity, as well as consideration of the logistics of mounting each type of study and the necessary outreach efforts; Paper 3 addresses how embedding three-arm random assignment sites in a conventional two-arm multi-site experiment permits calculation of the added value of specific treatment components. Paper 4 explores an ambitious partial factorial design that employs a very large sample allowing comparison of effects of different facets of a messaging and marketing strategy on increasing requests for services.
Each of these multi-armed designs yields nuanced information about relative benefits of treatment arm alternatives, resulting in a greater yield of information to guide policy decisions and evaluation questions. However, complexification of multi-armed designs does not make the basic challenges of basic multi-armed designed, reviewed at the start of the presentation, such as concerns with statistical power from multiple comparisons, go away. Therefore each panel member will balance discussion of costs and benefits, and how they address the problem. The fundamental focus will be on making clear the circumstances under which the specific multi-arm designs provide advantages and what the trade-offs are to their use, with the goal of providing useful direction for evaluation and policy learning.