1

Crowd-sourced labels don’t always reflect your target users. PAIR reweights training data so your model learns from the perspectives that matter most for your application.

AI models are only as good as their training data. This paper shows how 50+ years of survey science can help ML researchers collect better data - leading to fairer, more accurate models.

The order you show items to annotators matters. We found that changing the sequence of examples can shift labeling decisions - another reason to carefully design your annotation pipeline.

Small changes in how you ask annotators to label data can dramatically change your model’s behavior. We tested 5 versions of a hate speech labeling task and found significant differences in model performance.

How you design a labeling interface affects the labels you get. We show that task structure, ordering, and annotator backgrounds all shape training data quality.