Many policy studies estimate the impact of changes in state-level laws and regulations by using multi-state, longitudinal data. In these studies, researchers typically construct a panel of state-year observations and estimate a regression model, often with state and year fixed effects, to determine the association between a change in state policy and a relevant outcome. The policy change tends to be measured using a binary variable that is coded as one if the policy is present in a state at a particular time and zero otherwise.

When measured in this manner, the binary variable implies that the policy examined is the same in each state across time. However, the language of the law or regulation may vary from state to state or within a state over time in substantive ways that could change its impact. This variation would be obscured in models that measure the presence or absence of the policy with a zero-one variable.

To examine the extent to which use of a binary variable may contribute to erroneous conclusions about the impact of a policy, we used systematic searches to collect a sample of studies that incorporate this type of variable in multi-state research designs. Because our initial searches using select databases resulted in more than 1,500 articles, we narrowed our searches to articles published after January 1, 2010. Funding restrictions required us to further limit our searches to health-related policies. We next targeted the following topics that appeared most frequently in our initial searches: policies involving alcohol or tobacco and policies relating to contraception, infertility, abortion, and pregnancy. Within articles from the subsequent search results, we also examined references to identify additional studies for inclusion. Our final sample comprised 16 articles.

Of these articles, four described studies that used a range of measures to control for variation in the state policies examined. The remaining 12 articles described studies using a binary policy variable. We then researched the language and history of each state policy analyzed in these 12 studies to determine if the policies varied in substance between and within states over time. In only one of the 12 studies was the policy at issue uniform over time both between and within states. In the remaining 11 studies, we found substantive policy differences within and between states that we believe cannot be adequately captured by a zero-one policy variable. As a result, where these studies find that a policy has a significant impact on a particular outcome, the zero-one variable may underestimate that impact, capture the impact of some contemporaneous factor instead, or lead to spurious results.

Our findings suggest that when estimating the impact of policy changes, researchers should avoid using the zero-one policy variable in a multi-state, longitudinal study unless they can confirm the policy’s uniformity. Alternative approaches include single-state studies, use of ordinal variables where appropriate to capture policy differences, or multi-state studies that track differences in policies within and between states over time using multiple binary variables to capture unique features.