Association for Public Policy Analysis & Management

Panel: The Evidence Challenge for Public Management

(Public and Non-Profit Management and Finance)

Obtaining evidence for the effectiveness of management strategies with a traditional social-science approach is difficult. A RCT of a management strategy requires a treatment group that employs the strategy whose effectiveness is being tested, plus a control group that does not. Yet, a management “treatment” has few (no?) defined physical components; it is not a medical intervention, training program, or cash. A management strategy is essentially an “idea.” How can those conducting an RCT prevent the control group from employing this “idea”?
Suppose the New York Police Department had sought to obtain evidence of CompStat’s crime-fighting effectiveness with an RCT. NYPD could have randomly divided its 76 precincts into two groups: 38 precincts in the control group; 38 in the treatment group. Treatment-precinct commanders would participate in CompStat meetings, which could motivate them to reduce crime and help them learn effective tactics from other precincts. Control-group commanders would be excluded from these meetings.
Still, the control-group commanders could have learned what effective crime-reduction approaches were analyzed at these meetings, and what treatment-group commanders—responding to CompStat’s motivational stress—were doing to reduce crime. Thus, motivated precinct commanders in the control group—those who sought to be recognized as effective in reducing crime—could have adapted strategies that they thought might work in their precincts.
The possibility that control-group managers might learn and employ the treatment group’s strategies presents public managers (and public-management scholars) with a dilemma: What kind of evidence could be collected and analyzed to determine whether a management strategy—a management idea—is effective? Moreover, if a RCT revealed that those who employed a specific strategy produced better results than those who did not, what managerial instructions—based on what essential behaviors of the control-group managers—should be given to others to implement this proven strategy? What, exactly, is this management strategy that has “proven” “effective”?
After all, for a management strategy to be truly “proven” as “effective,” it has to “work” in multiple jurisdictions and policy fields. Yet, for public managers to intelligently apply an management strategy in different jurisdictions and policy fields—indeed, before a management strategy can be even tested—it must be carefully specified: It must be clear what the strategy is and what is it not.
Nevertheless, any manager who employs this proven strategy will need to adapt some of its operational specifics to his or her unique mission, organization, and political circumstances.
In response to this analytical challenge, this panel examines how to think about evidence for management strategies.
Musso analyzes a management strategy—Pay for Success/Social Impact Bonds—for which it is possible to obtain explicit evidence.
Van Ryzin considers how citizen inability/unwillingness to engage in the counterfactual thinking required to evaluate evidence may be reduced if they think the beneficiaries of the program are deserving.
Smith emphasizes that social-service agencies (particularly small ones using public and private funds) need to develop an ability to respond to new performance regimes.
Behn focuses not on explicit, operational knowledge but on the tacit knowledge of managing people that effective managers acquire through experience.