measurement error

Whether future AI models are fair, trustworthy, and aligned with the public’s interests rests in part on our ability to collect …

Crowd-sourced labels don’t always reflect your target users. PAIR reweights training data so your model learns from the perspectives that matter most for your application.

AI models are only as good as their training data. This paper shows how 50+ years of survey science can help ML researchers collect better data - leading to fairer, more accurate models.

The order you show items to annotators matters. We found that changing the sequence of examples can shift labeling decisions - another reason to carefully design your annotation pipeline.

5 minute lightening talk on methods to collect annotations for NLP models

As many in the AI field transition from a model-centric to a data-centric approach, the quality of training data is receiving increased …

People consistently overreport how much news they consume. We compared survey responses to passively collected behavioral data and found large gaps, highlighting the need to combine data sources for accurate measurement.