data quality

Data scientists are increasingly turning their attention to the collection of high quality training data. Those of us with expertise in data collection can apply lessons in web survey design to them.

Small changes in how you ask annotators to label data can dramatically change your model’s behavior. We tested 5 versions of a hate speech labeling task and found significant differences in model performance.

How you design a labeling interface affects the labels you get. We show that task structure, ordering, and annotator backgrounds all shape training data quality.