Why AI makes things up: hallucinations explained
The lawyer fined for citing six cases that never existed, the airline forced to honor its chatbot's invented policy, and why fluent machines fabricate — plus how to use them safely.
In May 2023, a New York lawyer named Steven Schwartz filed a court brief in a routine airline injury case. It cited six precedents, complete with case names, docket numbers, and quotes from the rulings. The opposing lawyers could not find any of them. Neither could the judge, because none of them existed. Schwartz had asked ChatGPT to do his research, and when he grew suspicious enough to ask it whether the cases were real, it assured him they were, and that they could be found in reputable legal databases. He and his colleague were fined 5,000 dollars, and the case became the standard warning told in every law office in the country.
The interesting part is not that a lawyer got lazy. It is that the machine did not malfunction. It did exactly what it is built to do, and understanding that is the single most useful thing to know about AI.
Why does a machine that knows so much invent things?
Because it does not know things the way you assume it does. A language model is trained to predict the next word so that the result reads like plausible human text. Legal briefs cite cases with certain name patterns, certain docket number formats, certain confident phrasing. When the model needs a citation and no real one is strongly burned into its patterns, it produces something with exactly the right shape. Not a lie — a lie requires knowing the truth and choosing against it. The model has no such checkpoint. Plausible and true are the same thing from where it sits.
Researchers call these fabrications hallucinations, which is a marketing-friendly word for a structural feature: the machine is a fluency engine, and fluency without verification produces confident nonsense at some unavoidable rate. Ask it for the plot of Hamlet and the training data carries it easily. Ask it for the middle name of a moderately known person, and it may fill the gap with statistical filler — smoothly, with no change in tone to warn you.
What has this actually cost people?
Small humiliations and real money. Air Canada's website chatbot invented a bereavement refund policy for a grieving passenger in 2022; when the airline refused to honor it, a Canadian tribunal ruled in 2024 that the company was responsible for what its bot promised and made it pay. Google's first public chatbot demo in February 2023 flubbed a fact about the James Webb telescope, and the market cut its parent company's value by around a hundred billion dollars that day. Professors have received essays citing books that were never written. News outlets have published AI-assisted articles recommending restaurants that do not exist. And more lawyers followed Schwartz than you would believe — courts around the world have now sanctioned dozens of filings with invented citations, years after the first famous fine.
The quieter cost is the daily one: people ask a chatbot a medical, legal, or financial question, get a fluent answer that is 90 percent right, and act on the 10 percent.
Is it getting better?
Yes, genuinely. Modern models hallucinate far less than the 2022 generation. The biggest fix was grounding: letting the model search the web or a document and then answer from what it found, citing sources you can click. When an answer is anchored to retrieved text, the invention rate drops sharply. Reasoning models that check their own steps help too, as does training that rewards saying "I do not know" — which older models almost never said, because the internet they learned from rarely says it either.
But better is not solved. The failure mode shrank; it did not disappear. And it hides best exactly where you are least equipped to catch it: topics you know nothing about, which is precisely when you asked.
How do I use AI without getting burned?
Use it like a brilliant, fast, occasionally overconfident intern — draft, summarize, brainstorm, explain, translate — and keep one rule: anything that must be true gets verified before it leaves your hands.
Ask for sources, then actually open them, because models can fabricate a citation as fluently as a sentence. Prefer tools that show what they searched. Be most suspicious of specifics — names, numbers, dates, quotes, URLs, case law, medication doses — because specifics are where plausible text and true text part ways. Never use "are you sure?" as verification; the model that invented the answer will happily invent the confirmation, as Schwartz learned. And when the stakes are real, treat the chatbot as the start of your work, not the end of it.
The machine is not broken and it is not lying to you. It is autocomplete wearing a very good suit. Once you stop expecting a librarian and start expecting a storyteller with an excellent memory, you will get enormous value from it and stop being surprised by the occasional beautiful, confident, completely invented answer.