Can People Detect an AI Mediator? A Pilot Turing-Test Study of AI-Assisted Conflict Resolution

Engineering notes

Engineering notes will be added by the aipentium editorial team.

Chinese explanation / 中文解读

中文解读待补充：本站会优先为大语言模型、生成式AI、ChatGPT相关技术、计算机视觉、深度学习等高价值论文补充中文说明。

Original abstract

Large language models (LLMs) are increasingly deployed in social and interpersonal contexts, yet little is known about whether people can identify an AI interlocutor in naturalistic, emotionally salient interactions. We report a pilot experiment in which 20 participants (10 dyads) engaged in a 60-minute text-based mediation of a simulated interpersonal conflict on Discord. Dyads were randomly assigned to a human mediator (control, n = 5 dyads) or to ChatGPT-4 acting as mediator (experimental, n = 5 dyads); the study was single-blind (participants were unaware that AI mediation was possible). After the session, each participant attempted to determine whether their mediator was human or AI. Identification accuracy did not differ from chance ($\chi^2$(1) = 0.20, *p* = .655), and Bayesian analyses indicated a general tendency to attribute mediation to a human agent rather than reliable detection of AI. Self-reported mediation effectiveness did not differ statistically between conditions in a two-sided test. However, equivalence testing did not support practical equivalence, and the observed effect pattern consistently favoured the AI mediator. Taken together, the results provide no evidence for human-mediator superiority and suggest a non-significant numerical trend favouring AI-mediated sessions (Hedges's *g* = −0.77, 90% CI [−1.50, −0.03]). Given the pilot nature and small sample size, these findings should be interpreted as preliminary and require replication in larger, preregistered studies.

Links and sources

Comments