Association for Public Policy Analysis & Management

Practitioners and evaluators are often interested in the effects of program components that are optional or based on encouragement. In this paper, we discuss a design to estimate a treatment effect of the main treatment component, the effect of the optional treatment component relative to the main component, and the total effect by using a three-group randomization design: one group is assigned to control, the next group only to the main treatment without the availability of the optional component, and the third to the main treatment plus the option and encouragement for the additional component. The analysis then uses assignment to encouragement for the optional component as an instrument of participation in the optional component. This design can also incorporate covariates in the analysis. Our paper then discusses sample size requirements, both for the main and optional treatment components as they relate to compliance. The paper also explores the benefits of including covariates in the analysis.

To estimate the effects of the main and optional components, the authors propose a three-group randomization strategy: one group is randomized into a control condition, a second group is randomized into a treatment condition with the main component only, and a third group is randomized into a treatment group that included both the main component and, additionally, encouragement for the optional component.

Of course, not all members of the third group that are encouraged to take up the optional component would likely do so, and so to estimate the treatment effect of the optional component relative to the main treatment, the authors propose an instrumental variable (IV) method to instrument uptake of the optional component with assignment to the third group.

This paper considers the estimation technique and the sample size requirements of this design both with and without covariates. Sample size formulas are presented for each of the three effects of interest:

1) the effect of the main component relative to control:

n = 2M²(1-R_W²) / δ₁²

2) the effect of the optional component relative to main component only:

n = 2M²(1-R_W²) / C²δ₂²

3) the total effect of the program relative to control:

n = [2M²(1-R_W²) / δ² ][1 + (1/C²) - (1/C)]

Where n is the number of observations in each of the three groups, δ is a Cohen’s d like effect size, C is the compliance rate with the optional component, and R² is the population squared correlation between the outcome and covariate. M is a factor based on the desired power and level of significance, which is about 2.8 for power of 0.8 and a two-tailed test with αlpha=0.05