Panel Paper: When You Say Nothing at All: The Surprising Predictive Power of Student Effort on Surveys

Saturday, November 8, 2014 : 3:30 PM
Galisteo (Convention Center)

*Names in bold indicate Presenter

Collin E. Hitt, Albert Cheng and Julie R. Trivitt, University of Arkansas
Policy research has yet to identify a consistent way to measure non-cognitive skills in children. This is problematic, given the fact that many education programs seek specifically to affect skills such as conscientiousness or grit – traits not well captured by test scores. For program evaluations, questionnaires attempting to assess personality traits are dogged by problems such as reference group bias, gaming, and student disinterest. We demonstrate that there are certain latent data within any survey dataset that can be used as proxy measures of non-cognitive skills. We posit that surveys are much like homework: repetitive, tedious, boring and well-within the cognitive abilities of most children to complete. The completion of a survey is a task. We calculate student effort on surveys within several prominent longitudinal datasets frequently used in social science research, such as the National Longitudinal Study of Youth 1997 and National Educational Longitudinal Survey 1988. Specifically, we calculate the frequency with which respondents provide incomplete, invalid or inconsistent answers. We then test whether survey-effort in adolescence is predictive of later life outcomes. Remarkably, the information captured in our measures of survey effort is a consistent predictor of educational attainment. We also examine outcomes such as income, crime, health and marriage. The pattern of relationships between survey effort and later outcomes is consistent with what one would expect of conscientiousness. Thus we suggest survey effort as a proxy measure of student skills, for use in program evaluations and public policy research.