Association for Public Policy Analysis & Management

Causal inference with observational studies requires researchers to estimate impact estimates conditional on a set of covariates that presumably removes all the selection bias. Given the absence of perfect theories about the data-generating selection or outcome models that would help in selecting covariates, researchers frequently try to remove selection bias by adjusting for a large set of covariates in a regression- or propensity-score-based analysis of the treatment effect. The underlying belief is that adjusting for a large set of covariates removes more selection bias than a small set of covariates. This is a misconception—a misconception that is regularly also held by methodologists and statisticians (e.g., Wakefield, 2013). As some authors already showed (Bhattacharya & Vogt, 2007; Pearl, 2010, 2011; Wooldridge, 2009), adjusting for a covariate may not reduce but instead increase bias. The covariate-adjusted impact estimate might be even more biased than the prima-facie estimate (without any adjustment for observed covariates). Our paper demonstrates that this happens not only in rare situations but much more frequently than commonly thought—even with highly correlated or unreliably measured covariates.

We thoroughly characterize the phenomena of bias-amplification and cancellation of offsetting biases in the presence of omitted variables. Using a simple linear (regression) setting with two confounders—one observed (X), the other unobserved (U)—that confound the causal relation between a treatment variable (Z) and outcome variable (Y), we investigate the conditions under which adjusting for X (but omitting U) increases instead of reduces selection bias. Overall, two phenomena are responsible for a potentially increasing bias: (i) bias amplification and (ii) cancellation of offsetting biases. Bias amplification occurs because the adjustment for X amplifies any remaining bias due to the omitted variable U. When the bias-amplifying effect dominates X’s bias-reducing effect, adjusting for X actually increases the bias in the impact estimate. Cancellation of offsetting biases is an issue whenever X and U induce biases in opposite directions such that their respective biases are perfectly or partially offset. Since adjusting for X cancels the bias-offsetting effect and then amplifies the increased bias, the adjusted impact estimate may be more biased than the prima facie effect. Whether adjusting for X results in an increasing bias depends on X’s correlation with U and its measurement reliability but also on the confounder type (i.e., whether X and U are outcome- or selection-related confounders that are highly predictive of the outcome or the treatment, respectively). Contrary to the general belief, we show that even with highly correlated confounders adjusting for X might still result in an increasing rather than diminishing bias. And measurement error in X does not imply that less bias is removed.

We discuss our findings using formulas, graphical models, and plots that visualize parameter combinations of increasing and decreasing biases. Finally, we present the results of a simulation with ten confounders of different type, indicating that even when eight of the ten confounders are included in the regression the bias in the impact estimate might still be larger than without any covariate adjustment.