Panel Paper: Reducing Bias In Observational Analyses of Education Data by Accounting for Test Measurement Error

Saturday, November 10, 2012 : 3:50 PM
Hanover B (Radisson Plaza Lord Baltimore Hotel)

*Names in bold indicate Presenter

J.R. Lockwood and Daniel McCaffrey, RAND

A common strategy for estimating treatment effects in observational studies using individual student-level data is analysis of covariance (ANCOVA) or hierarchical variants of it, in which standardized test scores are regressed on pre-treatment test scores, other student characteristics and treatment group indicators.  This class of models is commonly used in teacher value-added estimation.  Measurement error in the prior test scores, which typically is both large and heteroskedastic, erodes the ability of regression models to adjust for student factors and results in biased treatment effect estimates.  We develop a latent regression version of the ANCOVA model which provides estimators of treatment effects that are consistent for those that would be obtained if the prior test scores were based on infinitely many test items.  We demonstrate the effectiveness of the method for bias reduction compared to traditional ANCOVA in a case study of teacher value-added effect estimation using longitudinal data from a large suburban school district.