Order Effects in Annotation Tasks: Further Evidence of Annotation Sensitivity

Abstract

This paper provides further evidence of annotation sensitivity by examining order effects in annotation tasks. We show that the sequence in which annotators see items can systematically influence their labeling decisions, adding to the growing body of research on how data collection methods impact training data quality.

Publication
Proceedings of the 1st Workshop on Uncertainty-Aware NLP (UncertaiNLP 2024)