Association for Public Policy Analysis & Management

Student-level, school-based longitudinal evaluations often encounter study attrition when students move away from, or drop out of, study schools. In some cases, this mobility-induced attrition simply decreases sample size, weakening the study’s power to detect a treatment effect. At worst, attrition introduces bias into an otherwise well-designed study and threatens the study’s internal and external validity. This paper provides empirical benchmarks for the amount of attrition that can arise from natural rates of student mobility so researchers can plan for, and assess, attrition in studies that follow students over time. In particular, we addressed the following four research questions:

RQ1. To what extent do students move to different schools, districts, and states (or out of the public education system) over time, and how do these mobility rates differ across student characteristics?

RQ2. To what extent do mobility rates differ across educational settings and contexts?

RQ3. To what extent is attrition reported, and addressed, in impact evaluations?

RQ4. How do student mobility rates influence a study’s minimum required sample size (MRSS) to detect a given effect size?

Since applied researchers increasingly utilize data from district or state administrative data systems, the analyses distinguished between attrition that arises from students moving between schools within the same district, moving between schools within the same state, moving outside the state, and dropping out of the public school system.

For RQ1 and RQ2, we examined mobility for six different grade-level transitions using four longitudinal surveys from the National Center for Education Statistics. Overall, we found that the percentage of students with any type of mobility ranged from 12% (kindergarten to grade 1) to 46% (grade 1 to grade 5). Across the six transition periods examined, about 10% of students, on average, left their baseline school in a given year. Mobility rates for both elementary and high school grades suggest that mobility was more prevalent for African American students, low-socioeconomic students, students in single-parent families, and low-achieving students. Mobility was also more prevalent in schools that were located in cities, schools with higher percentages of underrepresented minorities, schools with higher percentages of students eligible for free or reduced-price lunch, and schools in larger districts.

For RQ3, we synthesized published results from 21 randomized evaluations that followed students for at least one school year after baseline. For these evaluations, the median attrition rate was 25%, and the interquartile range was from 16% to 36%. On average, studies lost 15% of their original sample for each year of follow-up.

For RQ4, we addressed a limitation in current power analyses by demonstrating how our attrition benchmarks can be used to adjust power analysis calculations for mobility-based attrition. This demonstration helps researchers assess how the MRSS changes based on student mobility and the data collection scheme (e.g., collecting outcome data in study schools versus collecting data from participating school districts). Examining mobility rates and power across data collection schemes will help researchers set more appropriate sample size targets, and weigh the trade-offs between data collection options and statistical power.