The Use of Propensity Score Methods for Addressing Attrition in Longitudinal Studies: Practical Guidance and Applications for Evaluating Early Childhood Interventions
*Names in bold indicate Presenter
As often results in the literature on estimation of program effects with non-experimental data, different propensity score methods for addressing attrition yield different results. Our novel approach to evaluating different methods involves relying on the estimation of a very early outcome in the Child-Parent Centers early intervention program (CPC, Reynolds et al., 2011) in which no attrition occurred. Following the general approach suggested by LaLonde (1986), we evaluate the performance of different methods for addressing attrition by comparing them to known results before attrition occurred. We remove from the sample observations that are missing outcome data in the long-term follow up, and we re-estimate the effects of the intervention on the earliest outcomes with these artificially reduced samples using different propensity score-based methods for correcting for attrition.
Our findings suggest that propensity score techniques - matching with different algorithms, stratification and inverse probability weighting- result in estimates very close to the true ones, suggesting that attrition bias may not be a concern in the CPC study. However, our results were sensitive to specification of controlled covariates in the propensity score estimation. Similar to findings for selection correction (Steiner, 2011), the use of many dimensions (e.g. individual, parental, demographic, home environment, neighborhood) and many variables for each dimension strengthen the results of the propensity scores estimation. Attrition from other studies or for other outcomes may turn out to be a greater problem.
This study aims to promote discussion among early childhood education researchers on challenges of estimating causal effects from longitudinal data when missing data is systematic. It also provides a practical guidance on basic questions like “when to use propensity scores techniques?”, “what type of information do we need to have?”, “how to test the assumptions of the model?”, “how to test the balance of treatment and comparison groups?” We propose strategies toward a more rigorous evaluation of early childhood interventions. Our approach has wide applicability to longitudinal analysis of any program or intervention in which attrition occurs but early outcomes exist for most or all of the sample.