Association for Public Policy Analysis & Management

Text-message based nudges and artificially intelligent chatbots have been shown to increase rates of timely college enrollment among college-intending high school graduates (Castleman & Page, 2015, 2017; Page & Gehlbach, 2018). Of course, matriculation to college does not guarantee students’ success. In fact, descriptive evidence underscores that many postsecondary enrollees never attain a degree. These non-completion rates are inequitably distributed across society. Among recent US high school graduates from low and middle SES backgrounds who access higher education, the modal outcome is no degree / credential eight years after high school completion (Kena et al., 2015). We test whether a text-message based conversational AI system can efficiently support enrolled college students to persist and succeed. In trying to boost the efficacy of these intelligent chatbots (e.g., Page & Gehlbach, 2017), we facilitate the personalization of our communication with students by integrating with our focal university’s student information system to customize the chatbot’s outreach to students.

We report on the use of this system at Georgia State University (GSU), a large, public postsecondary institution in Atlanta, GA. During the 2018-19 academic year, “Pounce,” the virtual assistant designed and implemented by AdmitHub (and named for the GSU mascot), sent text-based outreach to students regarding a variety of tasks and activities related to administrative processes, academic supports and on-campus engagement. To test the efficacy of the system, we implemented Pounce via a field experiment involving a sample of 7,580 undergraduates at a range of stages in their undergraduate trajectory. Some Pounce communications were directed to all students. Others, such as those related to issues including unpaid bills and registration holds, targeted only those students for whom the messages were relevant.

Our findings indicate that the outreach is particularly effective at raising student awareness of and responsiveness to acute challenges that would immediately hinder students’ enrollment status and progress. For example, among those students who began the semester with an outstanding bill, related text-based outreach increased by 9 percentage points (pp) (a 33% increase) the share who opened a case with the financial aid office and reduced by 10 pp (a 50% decrease) the share who had to withdraw. Likewise, for students with holds on their registration, the outreach increased by 8 pp (a 22% increase) the share who resolved all holds. Outreach related to less pressing tasks like meeting with academic advisors and refiling the FAFSA affected when students conducted these activities but did not increase levels of completion overall. Finally, students responded little to prompts about activities and topics that were not time sensitive and/or may have lacked relevance, such as outreach encouraging non-senior students to attend a graduate school information fair. Overall, the efficacy of this automated and scalable, yet personalized support was sufficient to cause a modest increase in students’ term GPA. Taken together, these findings help to inform the conditions under which text-based outreach may serve to improve student outcomes.