A Method for Annotating Dialogue Value Priority Based on Zero-Shot Chain-of-Thought

Value priority identification aims to uncover the implicit value priority attributes underlying a text, determining whether they align with specific values and their categories. This task is critical for detecting user language, evaluating content generated by large language models (LLMs), and exploring the ability of LLMs to assess human value priorities. However, due to the lack of datasets for human value priority identification in dialogue scenarios, research on modeling and identifying such priorities in conversations remains unexplored. Consequently, constructing a high-quality dataset for value priority identification in dialogues has become a pressing need.The creation of such a dataset poses significant challenges, as it requires annotators to possess substantial domain expertise, resulting in high annotation barriers. To address this issue, this study employs LLMs to annotate existing dialogue data, providing an annotated example of a value priority identification dataset in dialogues. This approach extends the application of LLMs in data annotation.Specifically, we propose a novel annotation methodology for dialogue value priority identification based on a Zero-Shot Chain-of-Thought approach, simulating human annotation results. Using this methodology, we construct a large-scale dialogue value identification dataset, ValueCon. Experimental results demonstrate the effectiveness of the proposed annotation method, as the ValueCon dataset outperforms manually annotated datasets in training and enhancing model performance.