Association for Public Policy Analysis & Management

This paper presents an effort to develop EWS for three Midwestern school districts. Student-level data from two cohorts of 8th and 9th grade students were used to identify early warning indicators for each district. The districts varied in size, demographic composition, and locale. Two of the districts serve large cities with populations greater than 250,000, while the third district serves a town that is between 10 to 35 miles from an urban area.  One of the urban districts has more than 40,000 students, while the other districts each have between 5,000 and 10,000 students. The percentage of students qualifying for free or reduced price lunch also varied, ranging from 41 percent to more than 90 percent of students. The four year graduation rates for the three districts ranged from 56 percent to 91 percent.

Candidate early warning indicators were gathered from student achievement, discipline, attendance, and coursework data.  Researchers evaluated the indicators according to how accurately each predicted whether students would fail to graduate high school on time.  In addition to providing the participating districts actionable information that will allow them to design their own locally validated early warning systems, this study offers insights into three research questions.  First, what points on the continuous data scales serve as thresholds for separating students who do and students who do not graduate on time?   Second, after identifying those optimal cut points, which indicators consistently identify at-risk students? Third, which risk factors are the strongest predictors of not graduating?

To answer the research questions, the research team analyzed the data provided by each of the three school districts. The team found the optimal cut point for each candidate indicator.  Next, the binary indicators were subjected to statistical tests to screen out indicators that were weak or unreliable predictors of non-graduation.   Indicators that passed the screening tests were designated as “robust”.  The researchers then determined the accuracy of the robust indicators—both as individual indicators and combinations of indicators—by examining the true positive rate, the false positive rate, and the predicted probability of not graduating. Findings show that the optimal cut points for classifying students as off track differ across districts and that certain indicators are more accurate predictors of non-graduation in some districts than others.  These findings underscore the importance of local validation to ensure indicators used in early warning systems accurately identify students who are at risk of not graduating on time.