Association for Public Policy Analysis & Management

Early warning indicators (EWI) or early warning systems (EWS) are becoming more prevalent in education as ways to inform educators on designing and implementing early interventions to help students before they fall too far off-track. Such systems compile a list of leading indicator variables to predict off-track status and use information from these indicators to identify candidate students for targeted supports. While there is a fast growing literature on the implementation, use, and effects of such systems, there has been little guidance on how to establish thresholds or cutpoints on the main leading indicators that can optimally separate students who are potential candidates for targeted interventions and those who are not.

This paper borrows approaches used in the field of medicine for clinical diagnosis and proposes a flexible approach to cutpoint identification in EWI/EWS that minimizes misclassification error. Receiver operating characteristic (ROC) curve analysis lays out two related concepts that operationalize misclassification: (1) selectivity (true-positive rate) and (2) specificity (true-negative rate). The study proposes several objective functions of selectivity and specificity that can identify cutpoints that optimally separate target and non-target individuals. Further, flexibility is added to the objective functions by adding parameters that account for relative costs of misclassification and prevalence of the outcome of interest. The result is an approach that can identify optimal cutpoints under different contexts.

I illustrate the optimization methodology in two examples using school district data. First, I identify optimal cutscores on kindergarten and first grade assessments to identify students in need of early literacy intervention. The approach leverages longitudinal data to identify the score profiles of students who exhibit later struggles with literacy. Next, using detailed student daily attendance data, I produce thresholds on daily attendance rates at different time points throughout the school year that can be used to identify students at risk of becoming chronically absent (missing at least 10% of the school year). The thresholds can be used to provide these at-risk students with targeted supports during the school year to prevent them from missing too much class time. The optimization approach here has the potential to be useful moving forward as districts continue to adopt and implement EWI/EWS.