Association for Public Policy Analysis & Management

This paper reports the final results from three-year randomized controlled trial of Achieve3000 in the Wake County Public School System (Wake County) in Raleigh, North Carolina. Achieve3000 is an early literacy program that differentiates non-fiction reading passages based on individual students’ Lexile scores. The driving force behind implementing Achieve3000 was the state’s 2012 Read to Achieve legislation, which mandated that students failing to meet proficiency standards in grade 3 reading must attend summer camp and pass a test before being promoted to grade 4. District staff were eager to launch a new program that could help increase reading proficiency among students receiving core instruction—as opposed to at-risk intervention—and ultimately reduce the share of children retained in grade 3 as a result of the new legislation. Achieve3000 was thus randomly assigned to 16 treatment schools for the 2013-14 school year, which included nearly 8,000 students in grades 2 through 5 and additional cohorts in subsequent years.

Our primary finding is that, on average, Achieve3000 produced small-to-moderate annual gains on select outcomes over three years of implementation. The outcome with the largest gains in the pooled sample (.09 SD) was the same one embedded in the software—an outcome used neither for state accountability purposes nor to drive instructional decisions. These particular results raise questions about the embedded nature of vendor assessments and whether student familiarity with the interface was associated with such gains. We also found that students in intact cohorts outperformed their control group counterparts over successive assessment waves on all three outcomes of interest, yet growth per wave was small and not substantively meaningful.

Our second major finding was that implementation varied widely by year and by school. Among the 16 schools to which Achieve3000 was assigned in the first year, 12 adopted the program and by year two, all schools used the program with varying levels of implementation fidelity. Achieve3000’s goal for implementation was for students to finish 80 or more activities by year’s end, which 24% of students achieved by the third year, a noticeable increase from 6% in year one and 10% in year two. This large increase in usage, however, did not translate into corresponding achievement gains on two of our three outcome measures.

These findings suggest that differentiated technology solutions like Achieve3000 may impact student achievement, but not necessarily for the outcome that matters most to district stakeholders. Our work contributes to the growing literature estimating the impact of technology on early literacy achievement (Kim & Quinn, 2013; Slavin, Lake, Chambers, Cheung, & Davis, 2009; Slavin, Lake, Davis, & Madden, 2011) and raises questions about the feasibility of districts meeting vendor usage targets and the relevance of their embedded assessments.