Association for Public Policy Analysis & Management

This presentation will discuss some of the challenges involved in estimating the impact of Jobs-Plus, a place-based employment intervention, on residents’ labour market outcomes, and an evaluation strategy for addressing them.  The challenges include, first, the issue identifying the outcomes of interest and how – and for how long – these might be observed.  While the programme focuses on improving residents’ employment and earnings and reducing their benefit receipt, an important consideration is the sustainabilityof these outcomes, raising the question of how long an observation period is needed to determine whether important effects will emerge and persist. 

 Second, there is the question of which methodological approach to use to estimate impacts of an estate-based intervention in a convincing way, when random assignment of individual residents to programme and control conditions is not feasible. This is influenced by practical considerations.  Ideally, a study would rely solely on randomly assigning a large sample of housing estates to deliver Jobs-Plus and others to carry on as usual.  With such a randomised control trial (RCT), a comparison of average outcomes would provide an unbiased estimate of impact.  However, the number of sites available to randomise may be too small to allow this approach to stand a reasonable chance of detecting effects at statistically significant levels, where they exist.  The evaluation design will therefore apply a comparative interrupted time-series (CITS) approach to the group-level randomised data.  A similar approach was used in the evaluation of the US Jobs-Plus programme.  It increases the statistical power of the analysis – that is, its ability to detect effects – by introducing additional modelling assumptions. 

The third issue follows from this: a CITS requires that outcomes be observed for a period of time both before and after the introduction of Jobs-Plus, for both the programme and control estates.  This can only feasibly be achieved through the use of administrative data.  Most suitable are the data held by the Department for Work and Pensions (DWP).  However, coverage with these data is best for those who have received a benefit at some point.  This is, therefore, the group for which the study will be able to estimate impacts, although it is also a group likely to make up a very large majority of residents of the targeted estates.

 This evaluation relies on the participation of social landlords and their ability (with partner organizations) to implement the intervention as intended.  Therefore, complementing the impact analysis, qualitative research with staff and residents and analysis of management information in Jobs-Plus sites will be conducted to help understand the nature and functioning of the intervention as it is actually operated. 

 The paper will summarize the strengths and limitations of this unusual impact design, and highlight the relevance of this approach for rigorously assessing impacts of other place-based interventions.