The more advanced artificial intelligence (AI) gets, the more it “hallucinates” and provides incorrect and inaccurate information.
Research conducted by OpenAI found that its latest and most powerful reasoning models, o3 and o4-mini, hallucinated 33% and 48% of the time, respectively, when tested by OpenAI’s PersonQA benchmark. That’s more than double the rate of the older o1 model. While o3 delivers more accurate information than its predecessor, it appears to come at the cost of more inaccurate hallucinations.
This raises a concern over the accuracy and reliability of large language models (LLMs) such as AI chatbots, said Eleanor Watson, an Institute of Electrical and Electronics Engineers (IEEE) member and AI ethics engineer at Singularity University.
“When a system outputs fabricated information — such as invented facts, citations or events — with the same fluency and coherence it uses for accurate content, it risks misleading users in subtle and consequential ways,” Watson told Live Science.
Related: Cutting-edge AI models from OpenAI and DeepSeek undergo ‘complete collapse’ when problems get too difficult, study reveals
The issue of hallucination highlights the need to carefully assess and supervise the information AI systems produce when using LLMs and reasoning models, experts say.
Do AIs dream of electric sheep?
The crux of a reasoning model is that it can handle complex tasks by essentially breaking them down into individual components and coming up with solutions to tackle them. Rather than seeking to kick out answers based on statistical probability, reasoning models come up with strategies to solve a problem, much like how humans think.
In order to develop creative, and potentially novel, solutions to problems, AI needs to hallucinate —otherwise it’s limited by rigid data its LLM ingests.
“It’s important to note that hallucination is a feature, not a bug, of AI,” Sohrob Kazerounian, an AI researcher at Vectra AI, told Live Science. “To paraphrase a colleague of mine, ‘Everything an LLM outputs is a hallucination. It’s just that some of those hallucinations are true.’ If an AI only generated verbatim outputs that it had seen during training, all of AI would reduce to a massive search problem.”
“You would only be able to generate computer code that had been written before, find proteins and molecules whose properties had already been studied and described, and answer homework questions that had already previously been asked before. You would not, however, be able to ask the LLM to write the lyrics for a concept album focused on the AI singularity, blending the lyrical stylings of Snoop Dogg and Bob Dylan.”
In effect, LLMs and the AI systems they power need to hallucinate in order to create, rather than simply serve up existing information. It is similar, conceptually, to the way that humans dream or imagine scenarios when conjuring new ideas.
Thinking too much outside the box
However, AI hallucinations present a problem when it comes to delivering accurate and correct information, especially if users take the information at face value without any checks or oversight.
“This is especially problematic in domains where decisions depend on factual precision, like medicine, law or finance,” Watson said. “While more advanced models may reduce the frequency of obvious factual mistakes, the issue persists in more subtle forms. Over time, confabulation erodes the perception of AI systems as trustworthy instruments and can produce material harms when unverified content is acted upon.”
And this problem looks to be exacerbated as AI advances. “As model capabilities improve, errors often become less overt but more difficult to detect,” Watson noted. “Fabricated content is increasingly embedded within plausible narratives and coherent reasoning chains. This introduces a particular risk: users may be unaware that errors are present and may treat outputs as definitive when they are not. The problem shifts from filtering out crude errors to identifying subtle distortions that may only reveal themselves under close scrutiny.”
Kazerounian backed this viewpoint up. “Despite the general belief that the problem of AI hallucination can and will get better over time, it appears that the most recent generation of advanced reasoning models may have actually begun to hallucinate more than their simpler counterparts — and there are no agreed-upon explanations for why this is,” he said.
The situation is further complicated because it can be very difficult to ascertain how LLMs come up with their answers; a parallel could be drawn here with how we still don’t really know, comprehensively, how a human brain works.
In a recent essay, Dario Amodei, the CEO of AI company Anthropic, highlighted a lack of understanding in how AIs come up with answers and information. “When a generative AI system does something, like summarize a financial document, we have no idea, at a specific or precise level, why it makes the choices it does — why it chooses certain words over others, or why it occasionally makes a mistake despite usually being accurate,” he wrote.
The problems caused by AI hallucinating inaccurate information are already very real, Kazerounian noted. “There is no universal, verifiable, way to get an LLM to correctly answer questions being asked about some corpus of data it has access to,” he said. “The examples of non-existent hallucinated references, customer-facing chatbots making up company policy, and so on, are now all too common.”
Crushing dreams
Both Kazerounian and Watson told Live Science that, ultimately, AI hallucinations may be difficult to eliminate. But there could be ways to mitigate the issue.
Watson suggested that “retrieval-augmented generation,” which grounds a model’s outputs in curated external knowledge sources, could help ensure that AI-produced information is anchored by verifiable data.
“Another approach involves introducing structure into the model’s reasoning. By prompting it to check its own outputs, compare different perspectives, or follow logical steps, scaffolded reasoning frameworks reduce the risk of unconstrained speculation and improve consistency,” Watson, noting this could be aided by training to shape a model to prioritize accuracy, and reinforcement training from human or AI evaluators to encourage an LLM to deliver more disciplined, grounded responses.
“Finally, systems can be designed to recognise their own uncertainty. Rather than defaulting to confident answers, models can be taught to flag when they’re unsure or to defer to human judgement when appropriate,” Watson added. “While these strategies don’t eliminate the risk of confabulation entirely, they offer a practical path forward to make AI outputs more reliable.”
Given that AI hallucination may be nearly impossible to eliminate, especially in advanced models, Kazerounian concluded that ultimately the information that LLMs produce will need to be treated with the “same skepticism we reserve for human counterparts.”