Do Large Language Models Agree on Cultural Annotation of Consumer Reviews

Engineering notes

Engineering notes will be added by the aipentium editorial team.

Chinese explanation / 中文解读

中文解读待补充：本站会优先为大语言模型、生成式AI、ChatGPT相关技术、计算机视觉、深度学习等高价值论文补充中文说明。

Original abstract

Identifying cultural and societal factors in large-scale consumer reviews (e.g., Amazon, Yelp, eBay) is critical for understanding customer behavior (Kim et al., 2023) and for informing effective business strategies based on user-generated content. However, manually identifying and labeling such factors for machine learning models is challenging due to the large volume of review data (Kaplan et al., 2020). Large language models (LLMs) are considered effective tools for accelerating the labeling of unstructured text (Ding et al., 2023); however, their consistency in capturing culturally grounded patterns in consumer reviews remains insufficiently understood. The existing approaches often rely on single model annotations, which limits the ability to observe variation across different systems and may introduce model-specific biases in interpretation (Ding et al., 2023). This study seeks to evaluate multiple generative models, including widely used systems such as ChatGPT, Gemini, and Claude, under a unified prompting framework for assigning structured cultural labels to consumer reviews. Differences will be analyzed in terms of inter-model agreement, labeling consistency, and the ability to capture multilingual and indirectly expressed opinions. In particular, the study aims to examine whether these models differ in interpreting indirectly expressed opinions and contextual cues embedded in consumer reviews. The findings of this study are expected to provide insights into the reliability and limitations of LLM-based annotation processes, highlighting the need for multi-model validation and careful evaluation when deploying large language models in socio-cultural annotation pipelines.

Links and sources

Comments