Publications

Crowd-sourced labels don’t always reflect your target users. PAIR reweights training data so your model learns from the perspectives that matter most for your application.

AI models are only as good as their training data. This paper shows how 50+ years of survey science can help ML researchers collect better data - leading to fairer, more accurate models.

The order you show items to annotators matters. We found that changing the sequence of examples can shift labeling decisions - another reason to carefully design your annotation pipeline.

Research apps allow to administer survey questions and passively collect smartphone data, thus providing rich information on individual …

Survey data producers increasingly provide estimates of nonresponse bias in several variables when they release or analyze data. …

People consistently overreport how much news they consume. We compared survey responses to passively collected behavioral data and found large gaps, highlighting the need to combine data sources for accurate measurement.

Motivated misreporting occurs when respondents give incorrect responses to survey questions to shorten the interview; studies have …

Small changes in how you ask annotators to label data can dramatically change your model’s behavior. We tested 5 versions of a hate speech labeling task and found significant differences in model performance.

Do survey respondents learn to game the questionnaire over time? We tested this in the U.S. Consumer Expenditure Survey and found the opposite. Responses actually improve in later waves.

Although counts of the novel Coronavirus (SARS-CoV-2) infections and deaths are reported by several sources online, precise estimation …

Filter questions are used to administer follow-up questions to eligible respondents while allowing respondents who are not eligible to …

How you design a labeling interface affects the labels you get. We show that task structure, ordering, and annotator backgrounds all shape training data quality.

We find rotation group bias in reports of spending in the U.S. Consumer Expenditure Survey. Contrary to our expectations, the more …

High response rates don’t always mean good data. Using seven rounds of the European Social Survey, we show that when interviewers help select who gets surveyed, high response rates can actually signal worse data quality.

Many surveys aim to achieve high response rates to keep bias due to nonresponse low. However, research has shown that the relationship …

Looping questions are used to collect data about several similar events, such as employment spells, retirement accounts, or marriages. …

Panel survey participation can bring about unintended changes in respondents’ behaviour and/or their reporting of behaviour. Using …

Administrative data are increasingly important in statistics, but, like other types of data, may contain measurement errors. To prevent …

How you structure filter questions in a survey affects the answers you get. We linked survey responses to administrative records and found that respondents answer strategically to shorten the interview.

Livestock are an important component of rural livelihoods in developing countries, but data about this source of income and wealth are …

Many face-to-face surveys use field staff to create lists of housing units from which samples are selected. However, housing unit …

This paper explores how well interviewers can judge which cases are likely to cooperate with a survey request and which are unlikely. …

The half-open interval procedure seems to offer an inexpensive method of reducing undercoverage in housing unit frames during data …

Web panels miss people without internet access. The LISS panel provided devices to non-internet households. We tested whether this extra effort actually reduced bias in research findings.

Using an experimental repeated listing design, this article demonstrates the presence of confirmation bias in dependent housing unit …