A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews

A study by Weixin Liang, Zachary Izzo, and colleagues estimates that between 6.5% and 16.9% of peer review text submitted to four AI conferences (ICLR 2024, NeurIPS 2023, CoRL 2023 and EMNLP 2023) could have been substantially modified by large language models. The authors use a maximum likelihood model that relies on expert-written and AI-generated reference texts to measure LLM use at the corpus level. The estimated share of LLM-generated text is higher in reviews reporting lower confidence, submitted close to the deadline, and written by reviewers less likely to respond to author rebuttals.