For the past two years, we’ve been constantly complaining that AI “talks nonsense” and warning people not to blindly trust what it says.
However, if you still hold that view today, it is highly likely that your understanding of the technology has already fallen behind.
To ensure you fully grasp this concept, I will do my best to explain it clearly using simple, everyday language.
01 | Large Language Models Are Fundamentally a “Probabilistic Prediction System”
If you wanted to summarize—in a single sentence—what current AI is actually doing, it really boils down to just one thing:
Based on a piece of text you provide, it guesses what should come next.
For instance, what would the AI say if you simply input the word “Congratulations”?
That depends entirely on the context.
At a wedding, the natural follow-up is usually “Happy Marriage”; at a baby’s hundred-day banquet, the odds favor “Congratulations on Your New Baby”; and if it were a victory banquet celebrating the national football team winning a championship, the only possible response would be “Champions!”
However, if you have absolutely no idea what kind of gathering you’ve stumbled into, the safest response is actually “Get Rich.” After all, weddings and births don’t happen every day, but everyone loves to hear about getting rich. If you greet everyone you meet with a hearty “May you get rich!”—and remember, “one doesn’t strike a smiling face”—you simply can’t go wrong.
The logic behind AI is actually just like this kind of social savvy: behind the scenes, it is calculating probabilities.
In its vast vocabulary, the probability of “Get Rich” appearing might be 60%, while “Happy Marriage” might be only 10%. Ultimately, it selects and outputs the word with the highest probability; when you see it, you think, “Hey, this AI actually speaks quite appropriately!”
Therefore, large language models are fundamentally a “probabilistic prediction system”—and this very nature provides the theoretical basis for why they sometimes “talk nonsense.”
Imagine you ask the AI to compose a poem with you—you provide the first line, and it provides the second. The result might look something like this:
You: Moonlight before the bed…
AI: Two pairs of shoes upon the floor.
You: Spring slumber, unaware of dawn…
AI: Mosquitoes biting everywhere.
Although the probability of such an occurrence is low, the possibility always exists. Even if it happens just once in a hundred attempts, you’ll still find yourself saying, “This AI really does talk nonsense!”
02 | Transforming the Erudite-but-Boastful “Scholar” into a Resource-Consulting “Expert”
When ChatGPT was first released in late 2022, it was, in essence, a contestant taking a “closed-book exam.”
At that time, GPT-3.5 relied entirely on the knowledge it had “memorized” during its training phase; it had absolutely no awareness of any events that occurred after its training cutoff date.
Consequently, when confronted with facts or topics it did not know, it would simply fabricate (or “predict”) an answer based on statistical probabilities.
The large-scale AI models of that era resembled an erudite—yet boastful—scholar who had ceased learning. If you asked it about something it knew, it would naturally provide a correct answer; but if you asked about something it didn’t know, it could only resort to “bluffing.”
As early as 2020, however, researchers at Facebook had already proposed a solution to this very problem: enable this erudite “scholar” to consult external resources whenever it encounters a question it cannot answer.
This concept evolved into the RAG architecture—a framework now widely adopted by large-scale AI models.
Simply put, the RAG architecture empowers an AI model to utilize search engines to look up information. When a user poses a question, the AI does not rush to provide an immediate answer; instead, it uses a search engine to retrieve relevant data. It then analyzes the retrieved information to enrich and strengthen its response capabilities, finally synthesizing the results into a coherent answer for the user.
Because this process involves three distinct stages—Retrieval, Augmentation, and Generation—the architecture was named “RAG,” derived from the first letters of these three steps.
It can be argued that the advent of the RAG architecture solved three major problems.
The first is the aforementioned issue of “spouting nonsense” (or hallucinating);
The second is that large-scale AI models no longer require costly, iterative retraining and fine-tuning; updating the underlying knowledge base is sufficient to ensure the model remains current with the latest information, thereby significantly reducing operational costs;
The third is the prevention of data leakage: companies can store their internal, confidential documents on private intranet servers, allowing the AI to generate responses based exclusively on these internal resources without the risk of sensitive training data being exposed to the public domain.
Final Thoughts
Of course, while AI models utilizing the RAG architecture no longer “spout nonsense” in the traditional sense, the information they consult is, ultimately, still sourced from the internet.
And the defining characteristic of the internet is precisely this: the truth is often indistinguishable from falsehood—a digital landscape where “nonsense” abounds.
Therefore, when encountering the nascent phenomenon of AI, we must maintain the habit of independent thinking and approach the answers it provides with a skeptical attitude; only then can we avoid being led astray.