Association for Public Policy Analysis & Management

As of 2015, 40 states have statewide Quality Rating and Improvement Systems (QRIS), and nearly all others are in the planning or piloting phases (QRIS National Learning Network, 2015). QRIS are accountability systems that aim to accurately measure program quality in early childhood settings and lead to program improvements through at least two mechanisms. The first is directly, by providing some combination of technical support or financial incentives tied to improvements in quality ratings.  The second is by providing parents with easy access to information about the quality of early childhood programs. This allows parents to “vote with their feet,” forcing lower quality settings to either improve or exit the market.

Despite substantial investment in early childhood accountability efforts, there is virtually no evidence on the extent to which such initiatives lead either to improvements in program quality or to changes in parents’ decisions about their children’s early learning experiences.  Our paper provides evidence on this high-profile policy initiative by examining North Carolina’s Star Rated License System, one of the most well-established QRIS in the country. We examine the causal effects of receiving a higher vs. lower QRIS rating on several observed dimensions of program quality as well as on enrollment and closures.

Our study addresses two research questions:

(1)  How do lower versus higher QRIS ratings impact subsequent measures of program performance (e.g., licensure; staffing levels and training, observational ratings)?

(2)  How do lower versus higher QRIS ratings impact the subsequent levels of program enrollments and program closures?

We use administrative data from North Carolina that includes all licensed care settings in the state from 2008-2014. To answer questions (1) and (2), we implement a regression discontinuity (RD) design, leveraging a “natural experiment” created by a feature of North Carolina’s QRIS. In particular, QRIS ratings are heavily influenced by a continuous observational measure of quality (the Environment Rating Scales, or ERS), such that small differences in ERS scores lead to substantially different likelihoods of receiving higher QRIS ratings. This allows us to isolate the true effect of receiving a higher vs. lower QRIS rating on future outcomes.

We find that centers that just missed the cut-off for a higher STAR rating improved their subsequent observational quality scores roughly 0.4 SD more than those who made the cut-off.  However, we find no differential effects of higher vs. lower STAR rating on other measures of quality (e.g. staff education/experience, child-staff ratios) or measures of take-up (program enrollment, % capacity filled, closures). Our study provides the first credibly causal evidence on some of the key mechanisms that underlie QRIS programs. In particular, it demonstrates that quality ratings can create incentives for centers to make measurable improvements.  Implications for the design of QRIS will be discussed.