Association for Public Policy Analysis & Management

Administrative data, such as school, hospital, and earnings records, provide an increasingly rich data source for conducting impact evaluations of interventions and programs. Obtaining these data, however, can be difficult due to data confidentiality concerns and limited agency staff availability for providing the data. One approach for facilitating access to these data is to minimize data requests for individual-level data and to instead request aggregate (grouped) data that can be used for impact estimation. This approach can complement other statistical methods such as masking the individual data while preserving its statistical properties (Matthews and Harel 2011).

This paper considers the required aggregate statistics to request from data agencies to rigorously estimate impacts for RCT and comparison group designs, and the statistical properties of various impact estimators that use the grouped data. The focus is on the estimation of average treatment effects and standard errors for full sample and baseline subgroup analyses (the typical confirmatory analyses). The paper considers aggregation methods for a full range of evaluation designs, including non-clustered designs where individuals are randomized and clustered designs where groups (such as schools, hospitals, or communities) are randomized. It also considers models with baseline covariates to improve precision, blocked designs, the inclusion of weights (e.g., to adjust for survey nonresponse), and the estimation of the complier average causal effect.

The paper develops methods for grouped data using design-based impact estimators that are derived using the building blocks of experimental designs with minimal assumptions (Freedman, 2008; Lin, 2013; Imbens and Rubin, 2015; Schochet, 2013, 2016, 2018; Yang and Tsiatis, 2001). As the paper shows, design-based estimators are conducive to using grouped data for a wide range of designs. The paper also considers similar methods for widely-used robust estimators to broaden the estimation options. The paper addresses the following research questions:

• What group-level (aggregate) statistics are required to fully reproduce the impact results based on individual-level data?
• What is lost in terms of bias and precision if the analysis is conducted using only group-level means on the outcomes, covariates, and weights but no other aggregate statistics (building on the work of Stoker, 1986)?
• How should the data be grouped to maximize precision, and what are tradeoffs regarding group size and the number of groups?
• To reduce data requests further, is it possible to conduct the subgroup analysis using only data on group-level mean outcomes and group-level subgroup proportions using ecological inference methods (King, 1997)?

The paper concludes by demonstrating key concepts using data from a large-scale RCT of the Job Corps program.

The paper fits directly with the APPAM conference theme on research to encourage innovation and improvement because it discusses new methods on how to facilitate the process of obtaining rich administrative data to estimate impacts using grouped data, which could hopefully help increase the conduct of rigorous impact studies to inform policy.