Panel Paper: Inverse Probability Weighting and Endogenous Subgroup Analysis in Social Experiment: Evidence from Simulation and Project Gate

Saturday, November 8, 2014 : 10:15 AM
Apache (Convention Center)

*Names in bold indicate Presenter

Ye Zhang, IMPAQ International, LLC
The main advantage of a large and well-executed social experiment is that the policy analysts can confidently rule out the possibility that unobserved differences between the treatment and control groups could explain the results of the study. In a world of heterogeneous treatment effects, experiments also make it possible to obtain unbiased estimates of treatment effects for subgroups. Treatment effects on subgroup are of particular interest to policymakers seeking to target policies on those most likely to benefit. As a general rule, subgroups must be created based on characteristics that are either immutable or observed before randomization so that they could not possibly have been affected by the treatment.

Many recent studies have addressed the issues related to subgroups determined endogenously through post-random-assignment event, such as dropouts from treatment group or substitution in the control group (Heckman, Hohnmann, Smith, and Khoo, 2000, Peck 2003, Bell and Peck 2013). Similarly, policy makers are interested in estimating how treatments affect those most in need of help, that is, those who would attain extremely unfavorable outcomes in the absence of the treatment. This is also an endogenous subgroup as treatment parameters of this nature depend on the joint distribution of potential outcomes with and without treatment, which is not identified by randomization (Heckman, Smith,and Clements, 1997).

A common approach for endogenous subgroup analysis is to use baseline characteristics to predict post-random-assignment endogenous subgroups through regression-based method, the approach creates symmetric subgroups between treatment and control groups, retaining the experimental design’s internal validity (Bell and Peck 2013). This paper develops subgroup identification and impact estimation methods based on recently developed inverse probability weighting (IPW) estimator, addressing functional form and common support concern associated with regression-based approach while producing better finite sample properties (sample size concerns). The theoretical analysis is supported by Monte Carlo simulation study where data generating process is known and a re-analysis of a large scale social experiment of entrepreneurship training program (Project GATE) where many treatment group members dropped out of the program and many control group members received similar trainings.