This presentation highlights the role of one or more waves of true pretest or proxy pretest data for reducing bias in observational studies when random assignment is not possible. True pretests are a pre-intervention measure of the outcome and proxy pretests are measures in the same conceptual domain as the outcome, but measured in a different form. The theoretical rationale for privileging pretests is that they are nearly always highly correlated with the outcome, and in many education contexts and also highly correlated with the selection process. Proxy pretests being highly correlated with true pretests will also be correlated with the outcome and selection process. It is these two correlations that ultimately determine how much selection bias will be reduced. More than one pretest time point can compensate for the unreliability of a single true pretest measure and help address the threats to internal validity associated with differential maturation.

Two empirical within study comparisons will be used to examine the extent to which an observational study that uses pretest and proxy pretest measures can produce trustworthy causal estimates. For the first within study comparison, the experimental data come from an a large-scale cluster randomized controlled trial (RCT) to study the effect of the state of Indiana’s benchmark assessment system on student achievement as measured on the state’s annual Indiana Statewide Testing for Educational Progress-Plus (ISTEP+) measures.  Fifty-nine K-8 schools volunteered to implement the formative assessment system in the 2008-09 school year. Thirty five of these schools were randomly assigned to implement the state’s formative assessment system. The remaining 24 schools were assigned to the control condition. The within study comparison will create a non-experimental comparison group drawing on data from students in K-8 schools statewide.

In the second within study comparison, experimental data are drawn from twenty five high schools in Washington State that participated in an RCT examining the effect of a new science curriculum during the 2009-10 and 2010-2011 school years. Schools assigned to the treatment condition received BSCS Science: An Inquiry Approach, year-long, comprehensive curricular materials, and teachers in these schools participated in a seven day professional development (PD) program.  The PD included a four day summer session and three follow up days of training spread across the school year. The comparison schools continued with the science curriculum and PD in place prior to the study. The quasi experimental comparison group will be drawn from high schools in the state of Washington that did not participate in the RCT.

In both cases, the data available for selecting a matched comparison group includes multiple waves of true and proxy pretest measures at both the school and student levels. The study will examine whether using all of these covariates allows for the replication of the experimental benchmark as well as the relative bias reduction associated with one and two waves true and proxy pretest data.