The Office of Evaluation and Research (OER) in New York City’s social service agency, the Human Resources Administration (HRA), promotes evidence-based policy and practice through research, evaluation, and policy analysis. One of the team’s primary responsibilities is to conduct small-scale internal program evaluations, and OER is increasingly interested in using random assignment design as the most effective—and in some instances, simplest—method for assessing impact. This paper describes a random assignment experiment that OER launched with the HRA’s Office of Child Support Enforcement (OCSE) in July 2011 to evaluate OCSE’s Cash Assistance Agreement Pilot (CAAP).

A primary goal of OCSE is to increase the rate of child support order establishment for children receiving cash assistance. One important barrier is that the majority of cases sent to court are dismissed—typically because the noncustodial parent (NCP) does not appear and is not successfully “served.” In 2010, more than 10,000 child support cases for children on cash assistance in NYC were dismissed for this reason. CAAP reflects the hypothesis that aversion to court—often described as adversarial, confusing, and intimidating—is an important driver of NCPs’ low appearance rate. The pilot offers an alternative process by which NCPs who are referred to court are also invited to OCSE’s Customer Service Center to discuss an agreement. If an agreement is signed in advance of the court date, the court will file an order without the NCP having to appear. If the NCP does not come to Customer Service or otherwise does not sign an agreement, the court date stands. OCSE hypothesized that CAAP’s alternative order establishment process would lead to more orders, quicker establishment, and increased order compliance.

This paper offers important insights into the benefits and challenges of implementing random assignment design—and internal program evaluations more generally—within a local social service agency. It begins by examining the process through which the evaluation design was proposed, approved, and implemented. It describes program administrator concerns—for example, that random assignment would unnecessarily complicate the pilot and pull too many cases out of the treatment process—along with the strategies used to address them. It also highlights the benefits of rigorous evaluation for clarifying program impact. For example, the treatment group’s order establishment rate was far higher than OCSE’s baseline estimate, and yet comparison to the control group demonstrates that the intervention did not in fact impact this outcome. Specifically, based on 1,257 cases referred to court during the pilot’s first year, there is no statistically significant difference between the treatment and control groups in order establishment rate (53.6% for treatment; 53.0% for control) or speed of order establishment (98 days for treatment; 93 days for control). Finally, the paper emphasizes the critical role of an implementation evaluation for understanding these results, and presents findings from a mix-methods process analysis that includes administrative data, observations at Customer Service, and interviews with staff (frontline and supervisors) and treatment group parents (custodial and noncustodial).