Association for Public Policy Analysis & Management

The Site-Specific Targeting (SST) program was a planned inspection program managed by the U.S. Department of Labor's Occupational Safety and Health Administration (OSHA). SST carried out enforcement actions to improve workplace health and safety where injury and illness rates were high. Enforcement actions included: (1) high-rate letters sent to workplaces to warn managers about their high injury-illness rates, and (2) inspections of worksites for compliance with safety and health regulations. The SST impact evaluation study was designed to measure whether SST enforcement actions affected: (1) regulatory compliance, measured by the probability of being cited for a violation during a follow-up inspection; and (2) health and safety, measured by the follow-up injury and illness rates. The study aimed to measure both the direct impacts of receiving letters and inspections and the indirect impacts of being assigned by OSHA to these enforcement actions. Indirect impacts allow detecting potential deterrent effects that fear of inspections may have had on regulatory compliance and health and safety. Two designs DOL CEO's initial contractor applied a randomized controlled trial (RCT) design to assess causal impacts of enforcement actions on regulatory compliance and health and safety. Implementation challenges diminished the statistical power of the RCT, limiting its ability to detect impacts of policy-relevant size. An existing injury-illness-rate inspection threshold, OSHA Data Initiative data, available inspections data, and the study team's matching algorithm for merging relevant OSHA datasets allowed the study team to implement an RDD with sufficient power at an acceptable additional cost. 2,520 worksites participating in the RCT were assigned at random to one of two treatments (i.e., high-rate letter alone or both high-rate letter and SST inspection) or to a control group. Treatments occurred in 2011, before DOL CEO replaced the initial evaluation contractor; outcomes were measured between 2012 and 2015. The RDD separated 7,045 worksites into "treatment" and "comparison" groups using 2006 injury-illness rate thresholds for OSHA enforcement actions, mimicking the 2008 SST program. Sites over the threshold became "treatment" sites; those under the threshold were "comparison" sites. This design allowed attributing observed differences in outcomes between groups on either side of the threshold to (1) OSHA's inclusion of worksites on treatment site lists and (2) actual receipt of a specific treatment. Although neither the RCT nor the RDD found statistically significant impacts, the project advanced methodologically how program evaluators outside the realm of K-12 education can apply RDD to existing administrative data to measure impacts. Using a secondary study design reinforced the weak but underpowered RCT impact findings, helped build knowledge about appropriate applications of quasi-experimental designs, and revealed some strengths and some weaknesses of both approaches.