annotation

Aligning NLP Models with Target Population Perspectives using PAIR: Population-Aligned Instance Replication

Crowd-sourced labels don’t always reflect your target users. PAIR reweights training data so your model learns from the perspectives that matter most for your application.

Stephanie Eckman, Bolei Ma, Christoph Kern, Rob Chew, Barbara Plank, Frauke Kreuter

Order Effects in Annotation Tasks: Further Evidence of Annotation Sensitivity

The order you show items to annotators matters. We found that changing the sequence of examples can shift labeling decisions - another reason to carefully design your annotation pipeline.

Jacob Beck, Stephanie Eckman, Bolei Ma, Frauke Kreuter