The Consortium for Educational Research and Evaluation–North Carolina evaluated North Carolina’s Race to the Top-funded STEM courses for underserved students offered through the state’s Virtual Public School (NCVPS) via a blended-learning model. Evaluation goals were to assess the extent to which this initiative contributed to: (a) the enrollment of underserved students targeted by the initiative; (b) the success of those students in the STEM courses offered; and (c) an increase in the availability of effective STEM teaching to students in high-need schools. This final summative evaluation report assesses program capacity, course quality, and program effectiveness using data collected during the first three semesters (Fall 2012 through Fall 2013).

Summative Findings

Capacity: The initiative reached nearly 400 students, including students from groups traditionally underrepresented in STEM fields. Data are not yet available to determine cost-effectiveness, but until NCVPS reduces the number of ongoing course revisions, significantly expands the number of LEAs with access to the courses, and increases the teacher-student ratio, the initiative is unlikely to be cost-effective.

Course Quality: Teachers, students, and independent reviewers all expressed concerns about course quality, both in terms of the rigor of the courses and their incomplete alignment to state standards. Most of the blended courses continue to under-utilize their e-format. However, strong communication between online and face-to-face teachers has led to significant contributions in course e-content, and as a result the online portion of the courses remains a prominent component. On most measures, the blended-course structure appears to have positively impacted student interactions with face-to-face teachers, though these interactions may be a product of smaller class sizes and not of blended learning. Much work remains to integrate the online teachers fully into the complete course experience and improve the quantity and quality of student-online teacher interactions. Opportunities for meaningful student engagement were moderate to high across all courses. Student evaluations of the courses have improved over time, with students highlighting in particular the smaller class sizes and certain aspects of the project-based learning approach.

Program Effectiveness—Preliminary Impacts on Students and Teachers: Formal test data were not available to assess quantitatively whether participating students grew academically, but analyses of student focus group and teacher interview data suggest that many participating students developed useful academic skills. Capacity-building among teachers was more evident in the second year of the program than in the first. Capacity-building continues to be a challenge for teachers new to the initiative, who often are overwhelmed by the challenges of converting to blended learning.

Recommendations

Implementers should: address ongoing concerns about course content, design, and delivery; reduce the number and complexity of program features; better integrate professional development; engage participating teachers earlier and involve them more in planning and design; provide balanced coverage for all aspects of STEM; formalize a participant feedback loop; design methods for supporting phased engagement of face-to-face teachers; and develop a longer-term initiative vision.