Panel:
Issues and Evidence in within-Study Comparisons
(Methods and Tools of Analysis)
*Names in bold indicate Presenter
Studies that compare non-experimental treatment effect estimates to an experimental benchmark are known in the literature as “within-study comparisons” (WSCs). This panel is comprised of four papers that contribute to this literature and fall into two broad categories: two of the papers use WSCs to move beyond estimation of the Average Treatment Effect (ATE); and the other two papers address issues of generalizability pertaining to WSCs. As such, the panel will present new evidence on the performance of non-experimental methods as well as solutions to known issues with WSCs.
The first two papers, by (1) Litwok and (2) Keller, extend WSCs beyond estimation of the ATE. Litwok focuses on estimating the Average Treatment Effect on the Treated (ATT) for a particular component of an intervention. He compares the performance of tools used for mediation analysis to an intervention that randomized access to a particular component of the intervention. Keller describes the design and implementation of a novel WSC that allows for experimental identification of conditional average treatment effects such as the ATT. Drawing parallels to Lalonde (1986), Keller focuses on using this WSC as a methodological testing ground.
The second two papers, by (3) White and (4) Wong, Steiner, and Anglin, address issues of generalizability related to WSCs. White proposes adding a second non-experimental analysis to incorporate the (often better) properties of the non-experimental estimate with respect to external validity into the WSC. Wong, Steiner, and Anglin use meta-analytic techniques to understand the conditions under which non-experimental methods come closest to replicating experimental benchmarks—that is, the conditions under which the WSC findings will generalize to other contexts.
WSCs are closely tied to the conference theme of engaging diverse perspectives on issues and evidence. The results of WSCs help researchers to gain perspective on the quality of evidence available in the absence of experimental variation. The papers in this panel, which all present novel contributions to the WSC literature, will both report the latest findings on these methods to the research community and stimulate discussion about how the literature can continue to develop.
The composition of the panel will bring together a variety of perspectives. The proposed panel is diverse and composed of both academic and non-academic researchers. The panel’s two discussants, Vivian Wong and Rob Olsen, have expertise tailored to the papers they will discuss and will stimulate a riveting discussion. Dr. Wong, who will discuss the first two papers, has authored seminal works in the WSC literature and continues to regularly publish. Dr. Olsen, who will discuss the second two papers, is an expert in generalizability and will focus his comments on the papers through that lens. Finally, the panel will be rounded out by the panel chair, Laura Peck. Dr. Peck has expertise in quantitative methods, particularly as related to experiments, and will be very engaging as a discussion moderator.
