Association for Public Policy Analysis & Management

There is increasing interest for state and local agencies to conduct low-cost experiments to test promising interventions and policies. The increased availability of administrative data provide a rich data source for such evaluations. However, state and local agencies experience barriers to planning these rapid cycle evaluations and analyzing the effects and costs of interventions. To directly address this need, the U.S. Department of Education, Office of Educational Technology (OET) has funded the development of a publically-available and free toolkit designed to reduce barriers to rigorous evaluation. The design of this toolkit has been heavily informed and supported by a technical expert working group that includes several leading practitioners and researchers from the education technology field.

The web-based, interactive Rapid Cycle Tech Evaluations (RCTE) toolkit is being field tested with districts and schools this fall (2016) for evaluations of the education technology products they use. The toolkit includes a set of tools and resources to support districts in carrying out all stages of the research process, from considering and prioritizing questions to conducting rigorous evaluations and interpreting the results. With a focus on educational technology, the toolkit guides users through an early-stage orientation process for better understanding different types of evidence and identifying the preconditions for undertaking a research project, followed by a dynamic planning and feasibility assessment process. To support the evaluation itself, the toolkit features several dashboards which enable users to conduct random assignment, propensity score matching, power calculations, and impact analyses. Finally, the toolkit offers guidance on the interpretation of results and provides resources to support public reporting and dissemination to build the broader evidence base for what works in education.

The RCTE toolkit is designed to make major strides in promoting the use of research evidence in education policy and school-level decision-making. The toolkit places the process and tools for rigorous evaluation into the hands of practitioners and creates natural opportunities for new partnerships with researchers.

This presentation will walk through the RCTE toolkit and outline the ways in which it is designed to facilitate the successful planning and implementation of experiments. The presentation will also include a demonstration of the toolkit’s evaluation tools for carrying out random assignment, propensity score matching, power calculations, and impact analyses.