Redefining Annotation Practices: Leveraging Large Language Models for Discourse Annotation

Abstract

Despite the promising results of large language models (LLMs)in labeling tasks, further exploration is needed to leverage them effectivelyfor linguistic data annotation. One of the most challenging tasks inthis regard is labeling discourse structures, which is highly subjective andoften involves ambiguity in class description. In this paper, we address thechallenge of using LLMs for hybrid annotation of the discourse structurein open-domain dialogues, relying on Eggins and Slade’s speech functiontheory. We conduct a comparative analysis between model-generatedannotations and human annotations, exploring the potential of LLMassistedannotation as a viable alternative to crowdsourcing.