This paper reports on a randomized controlled trial of two high school interventions where data on key postsecondary enrollment outcomes were collected for two phases. During the initial phase, outcomes were recorded from multiple sources, including administrative data and follow-up surveys. During the later phase, data came from administrative records only. The paper examines the reasons for disparities in levels of outcomes measured, analyses the sensitivity of impact findings to the data sources used and describes key considerations for future research design. 

The Future to Discover Pilot Project is a large-scale Canadian experiment designed to find out what happens when 5,400 incoming Grade 10 students are offered  a guarantee of a generous student aid grant or  three years of enhanced career education programming,  both, and  neither. Findings up to the sixth year following recruitment were based on combined survey and administrative data. Analysis at year six identified many significant impacts attributable to the interventions on access to postsecondary education.

The paper explains the consequences of the change in data sources for the reporting of impact estimates by reporting on a sensitivity analysis for the year of the switch in data sources (year six). It describes the contribution made by different features of survey and administrative data, and conventions for their analysis, to the determination of outcome measures, and ultimately estimation of impacts. The primary challenge is that administrative records in education are available only for those who participate in the outcome of interest within scope of detectionby a particular service provider. By contrast, surveys record outcomes regardless of participation and service provider. If administrative records cannot be located for a study sample member, his or her educational outcome is not known and researchers typically infer non-participation in the outcome of interest. This has direct consequences for the levels of the outcome of interest observed and may or may not affect the scale of impacts attributable to the interventions under test.

The change in data sources produced apparent drops in postsecondary enrolment rates that varied by subgroup and administrative data source. Nonetheless, these changes were orthogonal with respect to the assignment of treatment. Levels of impact with respect to postsecondary enrolment remained relatively stable.

The paper identifies four areas (coverage of outcomes, survey error, data matching practices, and conventions for the treatment of missing observations) that deserve particular attention when researchers have a choice with respect to data sources.

The study provides evidence that estimating program impacts in the context of a randomized experiment can be relatively robust to the data sources chosen. Internal validity and conclusions for policy need not be affected by changing data source even when it produces marked changes in levels of the outcome of interest observed.