Association for Public Policy Analysis & Management

We document the extent and nature of survey errors for program receipt and amounts for two transfer programs in three household surveys linked to New York State administrative cash welfare and SNAP program data. We focus on two sources of survey errors: misreporting and item non-response, examining the extent and nature of these two sources of errors and their implications for estimates of the determinants of program receipt and other statistics of interest. We do this for three surveys, the American Community Survey (ACS), the Current Population Survey Annual Social and Economic Supplement (CPS), and the Survey of Income and Program Participation (SIPP). Specifically, we first study the extent of misreporting. Our results confirm earlier evidence of high misreporting rates of program receipt. Households who fail to report one program are also more likely not to report true receipt of the other program. Mean errors in reported amounts conditional on reporting receipt are close to zero for SNAP, but large and negative for PA. However, the absolute value of the errors is high for both programs. We then examine the association of errors in reporting receipt, both false positives and false negatives, with respondent demographic characteristics like income, education and program receipt. We show that there are many statistically and economically significant determinants of both types of errors. Finally, we provide evidence on the consequences of these errors for the models of program receipt. In line with our results on the determinants of errors, we show that estimates of the individual determinants of program receipt such as income, education and race tend to be biased downward, often significantly so.

We then use our linked data to examine nonresponse and imputation as a source of survey error. We start by further documenting the high false negative rates and false positive rates among imputed observations in our three surveys. We find that item non-response is not conditionally random and imputation error is substantial and systematically related to covariates. Thus, neither including nor excluding imputed observations yields consistent estimates.