measurement error

Data scientists are increasingly turning their attention to the collection of high quality training data. Those of us with expertise in data collection can apply lessons in web survey design to them.

Small changes in how you ask annotators to label data can dramatically change your model’s behavior. We tested 5 versions of a hate speech labeling task and found significant differences in model performance.

Motivated misreporting occurs when respondents give incorrect responses to survey questions to shorten the interview; studies have …

Do survey respondents learn to game the questionnaire over time? We tested this in the U.S. Consumer Expenditure Survey and found the opposite. Responses actually improve in later waves.

Many large-scale government surveys do not have the budget or time to conduct experiments to assess the effects of questionnaire …

Motivated misreporting occurs when respondents give incorrect responses to questions to shorten a survey interview. Previous research …

As survey costs increase and response rates decrease, researchers are looking to alternative methods to collect data from study …

How you design a labeling interface affects the labels you get. We show that task structure, ordering, and annotator backgrounds all shape training data quality.

Filter questions are used to administer follow-up questions to eligible respondents while allowing respondents who are not eligible to …

We find rotation group bias in reports of spending in the U.S. Consumer Expenditure Survey. Contrary to our expectations, the more …