Hallucination
AI hallucination / the AI made it up / AI misinformation / confabulation
A hallucination is AI output that sounds right and is not. OWASP describes the model filling gaps in its training data using statistical patterns, without understanding the content, so the answer can be fluent and unfounded at once. Its own first example is an airline that was successfully sued over its chatbot.
OWASP files hallucination under a larger heading, misinformation, which it defines as an LLM producing false or misleading information that appears credible. Hallucination is named as one of the major causes. The mechanism is the useful part: hallucinations occur when models fill gaps in their training data using statistical patterns, without truly understanding the content. Nothing has gone wrong in the machinery. The gap was filled the way every other part of the answer was filled.
That is why fluency is no signal. The same process produces the sentences that are right and the sentences that are not, so they read identically. There is no tell.
OWASP names a second failure beside it. Overreliance is users placing excessive trust in generated content and failing to verify its accuracy, which turns a wrong answer into a wrong decision. Its first worked example is not hypothetical: an airline's chatbot misinformed travelers, and OWASP records that the airline was successfully sued as a result. What the assistant tells a customer is treated as what the business told them.
NIST supplies the standard that plausibility fails. It defines validation, borrowing from ISO, as confirmation through the provision of objective evidence that the requirements for a specific intended use have been fulfilled. Objective evidence is the phrase that matters, because a confident answer is not evidence of anything. NIST adds that validity for deployed systems is assessed by ongoing testing or monitoring, and that risk management may need human intervention where the system cannot detect or correct its own errors.
In practice
OWASP's remedies are not about better wording. It lists retrieving verified information from trusted sources during generation, cross-verification with human oversight and fact-checking for anything sensitive, automatic validation of key outputs in high-stakes settings, and labeling AI-generated content while telling users its limits. Every one of those is a process somebody has to own and run.
Not the same as
- A bug
- A bug is a repeatable defect that can be fixed. This is the system doing the thing it does, on a question where it had no material.
- Prompt injection
- That needs someone else's instruction to arrive in content the model reads. This needs no attacker at all.
Why it matters to you
An assistant on your site or phone line speaks in your name, and the airline case in OWASP's own list is what that costs when it is wrong. The question to put to anyone selling one is not whether it hallucinates, because every model can. It is what the answers are checked against, who notices when there is nothing to check against, and what the assistant says instead of guessing.
What to ask or check
- 01What does it check answers against, and what does it do when it finds nothing?
- 02Who reviews what it tells customers, and how often does that happen?
- 03Is AI-generated content labeled, with its limits stated where people will see them?
What people get wrong
That a hallucination is a malfunction that a better model or a stricter prompt will remove. OWASP describes it as gaps in training data being filled by statistical patterns, which is the same process that produces the correct sentences.
Red flags
- An assistant that answers everything, with no behavior defined for having no source.
- AI-generated answers going to customers with nobody reviewing a sample.
- A vendor who treats accuracy as a model property rather than a process you run.
Who owns it
You do, in practice. The airline in OWASP's example was held to what its chatbot said, and NIST puts validity on the deployer through ongoing testing and monitoring.
Where you will see it
Chatbots and AI receptionists answering questions about your prices, policies and availability.