Association for Public Policy Analysis & Management

This session will demonstrate features of the RCT-YES software package using a real-world dataset. The software was funded by IES at ED, and estimates average treatment effects (ATEs) for RCTs (or matched comparison group designs) to test the effects of interventions and policies. The software was developed to help facilitate the conduct of opportunistic experiments by states and local education agencies using administrative or other data sources, but can be used for impact estimation more broadly.

RCT-YES estimates impacts using design-based theory for a wide range of designs used in social policy research, including blocked and clustered designs. The software reports impact findings in formatted html tables that comply with industry standards, and employs several features to help minimize data disclosure risks. The software is free and can be run in R, Stata or SAS. The software uses a desktop interface application with formatted input screens to enter program specifications.   

The software conducts the following analyses:

• ATE estimation for continuous or binary outcomes for the full sample as well as population subgroups that are defined by pre-intervention (baseline) characteristics (moderator analyses).
• Simple differences-in-means estimators as well as estimators from regression models that adjust for baseline covariates to improve the precision of the ATE estimates.
• Standard error estimation and significance testing of the null hypothesis of a zero ATE against the alternative that it differs from zero, including multiple comparisons corrections.
• Estimators for (1) finite-population (FP) models where results are assumed to pertain to the study sample only (the default RCT-YES specification) and (2) super-population (SP) models where results are assumed to generalize outside the study sample to a broader population of similar students and schools.
• Estimators that incorporate weights to adjust for data nonresponse or other reasons.
• Methods to assess baseline equivalence of the treatment and control groups using baseline covariates.
• Estimates of the complier average causal effect (CACE) that pertains to compliers.

The presentation will demonstrate the software capability using data from a recent RCT in the social policy area, and will use the interface input screens to demonstrate how to input the program specifications for different types of designs and estimators.