Association for Public Policy Analysis & Management

Just before the Internet became an important part of college instruction, Paul Romer (1993) asked whether undergraduate students in a large introductory course of economics should attend class.  His results, and those of later papers, strongly suggested that attendance improved academic results. Over 20 years later the question has evolved:  In a digitally rich environment is face-to-face class time necessary?  Although the question has evolved, the challenge of obtaining unbiased estimates of the effect of classroom time remains formidable.   Just as attendance was likely correlated with motivation, conscientiousness, and ability, the decision to take a class online or in-person may depend on similar characteristics. Thus, comparisons of student performance in online or hybrid formats relative to traditional lecture classes must overcome the potential selection bias associated with the choice of format.

 To overcome such bias, recent studies have randomized students among online, hybrid and traditional classes. Despite their irreplaceable value, randomized experiments leave two important gaps. First, in the real world students choose their format. Do they sort themselves advantageously by matching their learning styles to the format that most improves—or at least does no harm—to their academic performance? Second, concerns about generalizability suggest the need for studies in many settings, fields of study and so on. The time, difficulty and expense of randomized experiments make observational studies an attractive alternative. How close can observational studies come to unbiased estimates of the effect of format? Which control variables are critical for removing omitted variables bias?

In this study we analyze the extent of selection bias in students’ choice of class format by comparing the performance of students who were randomized to a format with those who chose their format.  Specifically, we compare test scores of students in a traditional undergraduate lecture class of introductory microeconomics that met twice a week (150 minutes) with a hybrid or compressed format that met only once a week (75 minutes). Students in the both formats had access to the same online material of software and videos.   We randomized 756 students between formats in the fall of 2013.  In the fall of 2014, we offered the same course taught at the same times, in the same classrooms, by the same professors using the same book, software and lecture slides.  However, we let the 769 students enrolled in the course choose between a traditional and compressed format. We also surveyed them about factors that might explain format choice.

We find remarkably little evidence of selection bias. Differences in student performance between the traditional and compressed formats when randomized are only slightly greater in favor of the traditional class than when students choose their format.  Moreover, factors that predict choice of format, such as preferences for teacher and student interaction, do not explain student performance whereas strong predictors of student performance, such grade point average, do not predict choice of format.  Our results suggest that class time improves performance, but less so in an internet-rich environment than in earlier periods.