Text Classifications by Generative AIs Appear to Be Systematically Biased

Text classifications given by generative AI models can be systematically biased compared to human assessments, according to a recent study published on the ArXiv service.

The study compared classifications made by large language models, which are AIs that produce and understand human language, to previously manually made annotations. The comparison was based on data published by Boukes in 2024, where human coders had analyzed texts based on factors such as political content and style.

Researchers used several different generative large language models: Meta's Llama 3.1 model (8 billion parameter version), Llama 3.3 model (70 billion parameters), OpenAI's GPT4o model, and the Qwen 2.5 model (72 billion parameters). Each model was given five different prompts, asking them to classify texts according to five concepts: political content, interactivity, rationality, inappropriateness, and ideology.

The models achieved a "sufficient" performance when measured: the so-called F1 score, which combines precision and recall, was reasonable. Despite this, the machine classifications differed significantly from manual ones, especially when examining how much of a certain type of content was generally detected (prevalence). These differences also led to fundamentally different subsequent analyses when the results were later used in other examinations.

Additionally, generative language models resembled each other more than human annotations: the classifications of different models were more similar to each other than the agreement between them and manual classifications. Researchers emphasize that differences in F1 scores do not suffice to explain this bias, indicating a systematic bias independent of measured performance values.

Source: Are generative AI text annotations systematically biased?, ArXiv (AI).