Measuring the measurement error: a method to qualitatively validate sensitive survey data

Measuring the measurement error: a method to qualitatively validate sensitive survey data

Template G Content Blocks
Sub Editor

People may under-report sensitive and risky behaviors such as violence or substance abuse in surveys. Misreporting correlated with treatment is especially worrisome in causal analysis. We develop and test a survey validation technique that uses intensive qualitative work to check for measurement error in random subsamples of respondents. Trained local researchers spent several days speaking with and observing respondents within a few days of their survey, validating six behaviors: four potentially sensitive (crime, drug use, homelessness, gambling) and two non-sensitive (phone charging and video club expenditures). Subjects were enrolled in a randomized trial designed to reduce poverty and anti-social behaviors. We find no evidence of underreporting of sensitive behaviors, partly because (we discovered) stigma in this population is low. Nonsensitive expenditures were underreported, however, especially by the control group, probably because of strategic behavior and recall bias. The main contribution is a replicable validation method for observable, potentially sensitive behaviors.

June 11, 2014