Association for Public Policy Analysis & Management

Propensity scores techniques have become widely used to estimate causal treatment effects in the social sciences in the absence of randomized experiments or when instrumental variable estimation is not applicable.  While the literature provides guidance for applications of propensity scores approaches to correct for selection bias (Caliendo & Kopeining, 2005; Guo, Barth & Gibbons, 2005; Graham & Kurlaender, 2011; Steiner, 2011), it does not provide guidance on its application in the case of attrition bias.  Attrition bias may occur when there is differential loss of respondents between treatment and comparison groups.  Non-random attrition not only reduces statistical power but may also compromise the internal validity of policy estimates.  In the field of early childhood, long-term follow up has proved important in assessing the social return on investment (Barnett, 1995; Campbell et al., 2014; Gormley et al. 2011; Reynolds et al. 2011).  Although systematic attrition occurs, surprisingly few researchers attempt to reduce its bias.

As often results in the literature on estimation of program effects with non-experimental data, different propensity score methods for addressing attrition yield different results.  Our novel approach to evaluating different methods involves relying on the estimation of a very early outcome in the Child-Parent Centers early intervention program (CPC, Reynolds et al., 2011) in which no attrition occurred.  Following the general approach suggested by LaLonde (1986), we evaluate the performance of different methods for addressing attrition by comparing them to known results before attrition occurred.  We remove from the sample observations that are missing outcome data in the long-term follow up, and we re-estimate the effects of the intervention on the earliest outcomes with these artificially reduced samples using different propensity score-based methods for correcting for attrition.    

Our findings suggest that propensity score techniques - matching with different algorithms, stratification and inverse probability weighting- result in estimates very close to the true ones, suggesting that attrition bias may not be a concern in the CPC study. However, our results were sensitive to specification of controlled covariates in the propensity score estimation. Similar to findings for selection correction (Steiner, 2011), the use of many dimensions (e.g. individual, parental, demographic, home environment, neighborhood) and many variables for each dimension strengthen the results of the propensity scores estimation.  Attrition from other studies or for other outcomes may turn out to be a greater problem.

This study aims to promote discussion among early childhood education researchers on challenges of estimating causal effects from longitudinal data when missing data is systematic. It also provides a practical guidance on basic questions like “when to use propensity scores techniques?”, “what type of information do we need to have?”, “how to test the assumptions of the model?”, “how to test the balance of treatment and comparison groups?” We propose strategies toward a more rigorous evaluation of early childhood interventions. Our approach has wide applicability to longitudinal analysis of any program or intervention in which attrition occurs but early outcomes exist for most or all of the sample.