Association for Public Policy Analysis & Management

Impact studies of social programs are most useful when they directly inform key policy decisions, such as whether to cancel an existing metro-wide policy or to adopt a policy in a local neighborhood. An experimental impact evaluation uses random assignment to produce a reliable estimate of an urban policy’s effectiveness for the studied sample. However, the evidence from the studied sample does not necessarily generalize to the population for which policy decisions are to be made. Obtaining good policy guidance for the decision units that matter in major cities requires experimental research that is free of this “external validity bias.” Researchers need to translate evidence from a randomized experiment conducted in a specific place and sample into reliable policy guidance for other polities. Ongoing scholarship in the U.S. has tackled this challenge by seeking to understand the causes, consequences, and cures for external validity bias in randomized experiments. The current paper will take lessons from this body of research, as seen by one of its principal exponents, and apply them to metropolitan contexts in other parts of the world. Three specific questions will be addressed. First, how important is external validity bias empirically—that is, how big a distortion can it produce in urban policy decisions? Second, are there design or implementation strategies in conducting randomized experiments that minimize—or eliminate—external validity bias? Finally, how can data from a social experiment that lacks external validity be analyzed to produce findings with greater relevance to the population of policy interest?