# ChatGPT's Behaviour is Changing Over Time Source: https://arxiv.org/pdf/2307.09009.pdf Kiwi News: https://news.kiwistand.com/stories/ChatGPTs-Behaviour-is-Changing-Over-Time?index=0x64b7fe136621d2ad90d07bd84fcd66043ad6b353442106f5a9ad21011671587198622c6b Submitted by samchai.eth on 2023-07-19 UTC, 4 upvotes, 0 comments ## Summary Researchers Lingjiao Chen, Matei Zaharia and James Zou report that the behavior of GPT-3.5 and GPT-4 changed substantially between their March 2023 and June 2023 versions, which they say highlights the need for continuous monitoring of LLMs. GPT-4 identified prime versus composite numbers with 84% accuracy in March but 51% in June, which the article partly attributes to a drop in its adherence to chain-of-thought prompting. GPT-4 became less willing to answer sensitive and opinion survey questions in June, while its multi-hop question performance improved and GPT-3.5's declined. The authors also provide evidence that GPT-4's ability to follow user instructions decreased over time. ## Article How is ChatGPT's behavior changing over time? Authors: Lingjiao Chen, Matei Zaharia, James Zou GPT-3.5 and GPT-4 are the two most widely used large language model (LLM) services. However, when and how these models are updated over time is opaque. Here, we evaluate the March 2023 and June 2023 versions of GPT-3.5 and GPT-4 on several diverse tasks: 1) math problems, 2) sensitive/dangerous questions, 3) opinion surveys, 4) multi-hop knowledge-intensive questions, 5) generating code, 6) US Medical License tests, and 7) visual reasoning. We find that the performance and behavior of both GPT-3.5 and GPT-4 can vary greatly over time. For example, GPT-4 (March 2023) was reasonable at identifying prime vs. composite numbers (84% accuracy) but GPT-4 (June 2023) was poor on these same questions (51% accuracy). This is partly explained by a drop in GPT-4's amenity to follow chain-of-thought prompting. Interestingly, GPT-3.5 was much better in June than in March in this task. GPT-4 became less willing to answer sensitive questions and opinion survey questions in June than in March. GPT-4 performed better at multi-hop questions in June than in March, while GPT-3.5's performance dropped on this task. Both GPT-4 and GPT-3.5 had more formatting mistakes in code generation in June than in March. We provide evidence that GPT-4's ability to follow user instructions has decreased over time, which is one common factor behind the many behavior drifts. Overall, our findings show that the behavior of the "same" LLM service can change substantially in a relatively short amount of time, highlighting the need for continuous monitoring of LLMs.