The Data Must Flow

Data, rather than compute or energy, is the critical resource for training large language models, and AI companies face growing problems obtaining enough of it. GPT-4 was trained on an estimated 12 trillion tokens, and next-generation models are estimated to need up to 100 trillion, with most of the public internet already scraped. Copyright lawsuits against AI companies, including one by The New York Times, challenge how that data was used, and some startups are using crypto to address these data concerns.