Mit der Frage, ob und wenn ja inwiefern künstliche Intelligenz die deliberative Qualität in Diskursen erfassen und zu einer konstruktiveren Kommunikation in sozialen Medien beitragen kann, beschäftigt sich die Publikation von Patrick Frey, Prof. Dr. Cornelius Puschmann (ZeMKI, Universität Bremen), Smita Bhattacharya und Dr. Gregor Wiedemann (HBI). Die Forscher*innen aus dem Verbundprojekt ComAI evaluieren Deliberation in sozialen Medien mithilfe großer Sprachmodelle. Deliberation beschreibt dabei eine auf den Austausch von Argumenten angelegte Form der Entscheidungsfindung. Alle am Austausch Beteiligten sind dabei gleichberechtigt.
Die englischsprachige Publikation „Can AI Gauge Deliberative Quality? Evaluating Deliberation on Social Media With Large Language Models“ steht hier Open Access zum Download zur Verfügung. Sie ist in der Zeitschrift „Media and Communication“ erschienen.
Abstract
Although social media has become a central arena for deliberative discourse, debates are frequently characterized by incivility. Due to the volume of online discourse, it is challenging to assess when and where interventions, such as community-based counterspeech, are required. Accordingly, automated measurement of deliberative quality is a powerful tool for facilitating constructive online communication. Since deliberation is multidimensional, attempts to monitor online discourse must consider alternative forms of communication to foster inclusive deliberation. Whereas Type I deliberative dimensions, such as rationality and civility, are grounded in Habermasian ideals, Type II deliberation extends this perspective by incorporating alternative communicative forms, including emotionality and storytelling, into a more inclusive assessment of deliberative quality.
Addressing the limitations of prior studies that primarily focused on classic Type I deliberative dimensions and tested only a small number of large language models, this study evaluates multiple open-weight models to conduct a nuanced assessment of deliberative quality. We applied the large language model that achieved the best overall balance between predictive performance, model size, and computational efficiency to a dataset of 64,223 comments from German-language climate change discourse on X.
Our results demonstrate that large language models can reliably capture complex Type II deliberative dimensions despite class imbalances and subtle linguistic cues that generally hinder automated text classification. Class imbalance often impairs classification performance due to highly skewed label distributions, and Type II dimensions are particularly difficult to detect as they are often signaled by implicit, context-dependent markers.
The empirical analysis also reveals distinct communicative patterns: While public speakers predominantly employ civil, rational, constructive, and common-good-oriented communication, other debating users exhibit higher levels of reciprocal engagement, humor, and negative emotions, alongside more frequent incivility.
Our study demonstrates that AI-driven measurement can serve as a foundational tool for identifying norm violations, directing counterspeech toward specific debates, promoting inclusive digital public spheres, and observing broader long-term effects of moderation and interventions on the quality of debate.
Frey, P., Bhattacharya, S., Wiedemann, G., & Puschmann, C. (2026). Can AI Gauge Deliberative Quality? Evaluating Deliberation on Social Media With Large Language Models. Media and Communication, 14, 12494. https://doi.org/10.17645/mac.12494