Association for Public Policy Analysis & Management

Roundtable: Building Transparency & Reproducibility into Federal Evaluation

(Methods and Tools of Analysis)

Evaluation of Federal programs and policies can be strengthened by a greater focus on transparency and reproducibility. Transparency in methods, detailed reporting about policies or interventions being evaluated, pre-commitment to specific methods for data collection and analysis, and making data publicly available whenever possible all help to ensure that evaluation results are reliable and actionable — and several of these transparency-related practices are specifically highlighted in the President’s Budget for Fiscal Year 2020.

With the recent enactment of the Foundations for Evidence-Based Policymaking Act, it is timely to consider the unique challenges and opportunities associated with building transparency and reproducibility into Federal evaluation. This roundtable will bring together a variety of perspectives on this issue. Participants will discuss:

• the importance and benefits of transparency and reproducibility for supporting evidence-based policy (including generating stronger evidence and laying the groundwork for subsequent reevaluations of the same program or policy);
• methods for ensuring transparency and reproducibility that can be built into workflows for evaluation (for example, the workflow developed by the Office of Evaluation Sciences at the U.S. General Services Administration);
• building  transparency and reproducibility into grants and contracts for evaluation (as developed, for example, by the Office of Planning, Research & Evaluation within the Administration for Children and Families at HHS); and
• how to support policymakers and practitioners in understanding what transparency and reproducibility mean for the strength or reliability of different studies or pieces of evidence (for example, by revising standards of evidence for Federal evidence reviews and clearinghouses and tiered-evidence criteria).

Participants will represent a variety of perspectives, including:

• the perspective of the Federal evaluation community and a high-capacity Federal office with a formal evaluation policy,
• the perspective of a Federal evaluation office that has developed a detailed workflow for conducting evaluations of interventions with other Federal agencies,
• the perspective of researchers developing tools and evidence standards related to transparency and reproducibility of evaluations, and
• the perspective of a contractor who conducts evaluations on behalf of Federal agencies.

Panelists will describe challenges they face in their various roles and promising approaches they have developed to address these challenges, as well as offer more general perspectives on building transparency and reproducibility into Federal evaluation. This session will be of interest to those who carry out evaluations, those who advise policymakers on using the results of evaluations, and those who shape higher-level policies and processes for Federal evaluation offices.