Association for Public Policy Analysis & Management

The authors present a quasi-experimental approach for providing lower- and upper-bound estimates of true program impact by using selection bias theory and variability in program dosage. A corporate foundation funded 18 high-school completion programs across the United States. The impact of each was estimated annually, on grade-point average, school attendance, and on-track to graduation. Propensity score matching (PSM) was used to create non-program comparison groups. The number of days of program participation is used to proxy the program dosage delivered to each student. The PSM impact estimates are likely negatively biased, representing a lower-bound estimate of true program impact. Program students were selected based on risk factors which varied by program, most of which were unmeasured (e.g. in foster care) and likely pull down outcomes of program participants relative to comparison group members. Variance in program dosage allows for a different estimation approach, assuming higher dosage causes more positive outcomes. High dosage students are likely to have unmeasured supports (e.g. motivation, supportive caregivers) that lead to better outcomes independent of high participation. Thus a dosage-based estimate is likely to be positively biased, representing an upper bound on true impact. Application of these methods across 54 outcomes reveals that 43 follow the predicted pattern. PSM impacts are smaller than the dosage-based impacts. In nine cases, the lower bound is higher than 0 and statistically significant, providing unambiguous policy conclusions.