Association for Public Policy Analysis & Management

In recent years, researchers and policy makers have successfully applied insights from behavioral science to generate behavior changes. The usual approach in behavioral economics is to identify the intervention or ‘nudge’ that has the largest average effect, which is a function of the degree to which the relevant behavioral bottleneck is present and the responsiveness of people to the corresponding nudge. This intervention is then administered to everyone. In principle, we could achieve even larger gains if we were able to identify subgroups that experience different behavioral bottlenecks and then to tailor nudges to address these.

These heterogeneities in responses have proven difficult to identify in practice because of the risk of false discovery, and because usual statistical adjustments reduce statistical power. However, machine learning techniques can provide rigorous ways to test for heterogeneity across numerous potential dimensions without concerns about false discovery. If there are heterogeneities, this could lead to developing a new sort of ‘personalized nudge,’ following on the same sort of logic that motivates much of the movement towards ‘personalized medicine.’

We test these insights to a large-scale policy experiment that seeks to reduce failures to appear in court (FTA). FTA is a prevalent problem: in 2014, nearly 40% of individuals issued a ticket for a minor violation in New York City did not show up to court, and were issued an arrest warrant as a result. We designed a large-scale randomized controlled trial (RCT) in New York City to evaluate the effectiveness of the behaviorally-informed text messages reminders to reduce FTA. We find that relative to a “no-message” control group, people receiving text messages are 15% less likely to FTA. On average, messages that highlight the negative consequences of FTA are more effective than messages that help people make a plan to attend court (19% vs. 11% reduction in FTA).

However, among those who FTA in NYC, there could be a variety of motivations and factors contributing to that behavior. While on average less effective, messages that do not highlight the negative consequences of FTA could be more helpful for some defendants. We follow Athey and Imbens (2015) and Wager and Athey (2015) to use machine learning (specifically, causal trees and causal forests) to identify heterogeneous treatment effects, across treatment arms, and across recipient characteristics. Based on these results, we are designing a follow-up RCT to test the effectiveness of personalized messages, compared to the single-best message to everyone.