The Perils of AI Model Collapse

As we navigate the evolving landscape of AI, it's becoming increasingly clear that traditional search methods are struggling to keep up. With major players like Google leaning heavily into AI, one might expect improvements. However, the reality is a bit more complicated.

In recent months, I've observed a troubling trend: AI-enabled searches are often yielding less reliable results. When seeking hard data—like market-share statistics—the outputs frequently originate from dubious sources. Instead of the authoritative figures found in 10-K filings from the SEC, I receive vague summaries that resemble the truth but miss the mark.

This isn't unique to one platform; I've tested various AI search tools, and they all seem to produce these questionable results. This phenomenon is known as Garbage In/Garbage Out (GIGO), or more formally, AI model collapse. This occurs when AI systems, trained on their own outputs, gradually lose accuracy and reliability, leading to what's been described as a 'poisoned' model.

Model collapse stems from three main factors: error accumulation, loss of tail data, and feedback loops. Each new model generation inherits flaws from its predecessors, which causes outputs to drift away from the original data patterns. As rare events are erased from training datasets, concepts become blurred, leading to a lack of diversity in outputs.

Interestingly, a recent Bloomberg Research study examined Retrieval-Augmented Generation (RAG), a method that allows large language models (LLMs) to pull information from external sources. While RAG has the potential to reduce hallucinations, it ironically increases the risk of leaking private data and producing misleading analyses. Amanda Stent, Bloomberg's head of AI strategy, highlighted the implications of this finding, stressing the need for responsible use of RAG in everyday applications.

Yet, the notion of a 'responsible AI user' feels somewhat contradictory. Despite the promise of AI enhancing productivity, the reality often sees users resorting to generating subpar work—ranging from students crafting mediocre reports to businesses prioritizing profit over quality. The drive for operational efficiency can sometimes overshadow the importance of quality content.

As we continue to invest in AI, there may come a point where model collapse becomes too significant to ignore. If we consider OpenAI's claim of generating around 100 billion words daily, it raises concerns about the quality of information flooding the internet. The implications of this are profound, and as I see it, we may already be witnessing the early stages of this decline.

Cited: https://www.theregister.com/2025/05/27/opinion_column_ai_model_collapse/