Association for Public Policy Analysis & Management

This presentation describes how a multi-armed randomized experiment with a partial factorial design and 19 study arms was used to test multiple variants of a behaviorally-informed marketing strategy. In particular, we tested whether behavioral insights could be used to increase demand for a safety consultation service offered by the U.S. Occupational Safety and Health Administration. We used a brochure-based approach incorporating behaviorally informed messages and other insights, and sought to determine which variants of this marketing were particularly effective. We tested multiple variants related to: (1) the content of the marketing, in particular new messages designed to appeal to various motivating factors and based on psychosocial theories of behavior change; (2) the presentation of that content using various formats or “exemplars”; and (3) sending an accompanying email, which was intended to remove minor barriers to responding.

Our experiment used a partial factorial design with 19 study arms and a very large research sample—97,182 establishments—to test the impact of each message, format, and delivery mode compared with an existing (not behaviorally informed) informational brochure and a no-marketing counterfactual. Our main research goal was to determine whether behaviorally informed marketing was effective at increasing requests for OSC services. A second research goal was to predict the impact of the most successful marketing strategy (i.e., combination of message, format, and mode) so that OSHA would know what to anticipate if that strategy were implemented at scale. We used two related (but distinct) methods to address these two goals. Both begin with a common mixed (i.e., fixed and random effects) ANOVA model. We addressed the first research goal primarily from the fixed effects; we addressed the second research goal by calculating best linear unbiased predictions (BLUPs) from the full mixed model, where the BLUP involves “shrinkage” as in empirical Bayes (EB) approaches. This presentation will discusses the strengths and limitations of both approaches.

Our findings show that marketing via brochures was effective overall, nearly doubling the rate of requests for services, but that the behaviorally informed materials performed no better than OSHA’s existing informational brochure. Because the regression model fits the data well (i.e., the coefficients are precisely estimated and the model has good explanatory power), the EB predictions for each of the marketing strategies tended to be very close to the model-based predictions (i.e., they are close to the impact estimates calculated by adding together the coefficients from the regression model).

The findings from this study highlight the power of a factorial design for addressing questions about which of several program variants are most effective.