# 33-46% of MTurk crowd workers used LLMs when completing a task

Source: https://arxiv.org/abs/2306.07899
Kiwi News: https://news.kiwistand.com/stories/33-46percent-of-MTurk-crowd-workers-used-LLMs-when-completing-a-task?index=0x648a72f6ea8a7411cf42f7bd96a361df4059e00d3c934df2d6a9e93f25305992790d9208
Submitted by volpe.eth on 2023-06-15 UTC, 1 upvote, 0 comments

## Summary

Researchers Veniamin Veselovsky, Manoel Horta Ribeiro and Robert West report that many crowd workers likely used large language models when completing text tasks on Amazon Mechanical Turk. In a case study, they reran an abstract summarization task from the literature and, combining keystroke detection with synthetic text classification, estimated that 33-46% of workers used LLMs. The authors note that generalization to less LLM-friendly tasks is unclear, and they call for platforms, researchers and workers to find ways to keep human data human.

## Article

Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks

Authors: Veniamin Veselovsky, Manoel Horta Ribeiro, Robert West

Large language models (LLMs) are remarkable data annotators. They can be used to generate high-fidelity supervised training data, as well as survey and experimental data. With the widespread adoption of LLMs, human gold--standard annotations are key to understanding the capabilities of LLMs and the validity of their results. However, crowdsourcing, an important, inexpensive way to obtain human annotations, may itself be impacted by LLMs, as crowd workers have financial incentives to use LLMs to increase their productivity and income. To investigate this concern, we conducted a case study on the prevalence of LLM usage by crowd workers. We reran an abstract summarization task from the literature on Amazon Mechanical Turk and, through a combination of keystroke detection and synthetic text classification, estimate that 33-46% of crowd workers used LLMs when completing the task. Although generalization to other, less LLM-friendly tasks is unclear, our results call for platforms, researchers, and crowd workers to find new ways to ensure that human data remain human, perhaps using the methodology proposed here as a stepping stone. Code/data: this https URL