Association for Public Policy Analysis & Management

Deep reinforcement learning using convolutional neural networks is the artificial intelligence (AI) technology behind autonomous vehicles, algorithmic medical diagnostics, and Facebook automated photo tagging. Although AI often performs technical tasks faster and better than people, it is less clear whether AI could substitute for human judgment in addressing individual needs. We investigate this possibility in the context of students’ transition from high school to college. Even after acceptance into college, students must navigate several well-defined but challenging tasks, such as completing the FAFSA, submitting transcripts, obtaining immunizations, accepting loans, and paying tuition, among others. Without support, students can succumb to “summer melt,” the phenomenon where college-intending students fail to matriculate (Castleman & Page, 2014). Previous efforts to address summer melt have supported students with additional individual counselor outreach (Castleman, Owen, & Page, 2015) or through automated, customized text-message based outreach (e.g. Castleman & Page, 2015). Under both strategies, students could communicate with advisors one-on-one. Both approaches improved on-time college enrollment; however, scaling would require significant resources because of the need for a counselor to staff all follow-up communication. AI is a potential solution.

We test whether a conversational AI system can efficiently support would-be college freshmen with the transition to college through personalized, text message-based outreach over the summer. We report on the use of this system at Georgia State University (GSU), a large, public postsecondary institution in Atlanta, GA. During summer 2016, “Pounce,” a virtual assistant designed and implemented by AdmitHub (and named for the GSU mascot), sent customized text messages, based on students’ progress on required tasks recorded in the university’s information system, to admitted students. To test the efficacy of the system to help students complete required pre-enrollment tasks and matriculate by the fall, we implemented Pounce via a field experiment. At the study’s outset, some admitted students had already committed to GSU, while others were still choosing (or had committed elsewhere). Consequently, we hypothesized that Pounce would function differently for these two groups. We stratified our sample and randomization accordingly.

The intervention had a significant, positive impact on GSU-committed students. GSU-committed treatment students were 3.3 percentage points more likely to enroll, a 21 percent reduction in summer melt. Consistent with the theory of action underlying summer melt interventions more broadly, the outreach improved students’ success with accessing financial aid, submitting required paperwork and attending orientation, among other requirements.

References

Castleman, B. L., Owen, L., & Page, L. C. (2015). Stay late or start early? Experimental evidence on the benefits of college matriculation support from high schools versus colleges. Economics of Education Review, 47, 168–179.

Castleman, B. L., & Page, L. C. (2014). Summer melt: Supporting low-income students in the transition from high school to college. Cambridge, MA: Harvard Education Press.

Castleman, B. L., & Page, L. C. (2015). Summer nudging: Can personalized text messages and peer mentor outreach increase college going among low-income high school graduates? Journal of Economic Behavior and Organization, 115, 144–160.