What is hallucination?

A hallucination is a confident answer that is false or unsupported by its source: an invented refund window, a discount code that never existed, a feature the product does not have. The model is not lying. It is doing what it was trained to do, producing plausible text, and plausible is not the same as true. NIST's generative AI profile from July 2024 calls this confabulation and treats hallucination as the colloquial name.

It matters in support more than in most uses, because the customer acts on the answer. A chatbot that promises a refund creates a refund dispute, and in 2024 a Canadian tribunal held an airline to a discount its chatbot had described wrongly. Hallucinations also hide well: they are fluent, specific and polite, so a review that reads only for tone will miss them.

Where the idea runs out: no setting removes hallucination from a language model. What works is restricting what the model may answer from, called grounding; making it say it does not know when retrieval finds nothing; keeping facts out of instructions and in the knowledge base; limiting what tools it can run on its own; and reading a sample of real answers every week. A rising fallback rate is often the cheaper problem, because an admitted gap can be fixed with a page and a confident wrong answer may never be found.