Why do LLMs say things that are not true?
An LLM (large language model) does not look up facts to answer; it builds sentences by choosing words that are likely to come next. So even about things it does not know or remembers only vaguely, it produces plausible answers in a confident tone. This is called hallucination.
Hallucination does not go away when you change the model or the settings. What you can do is ask in ways that make it less likely and catch the errors that do occur. Knowing where it tends to happen tells you what to check and cuts the work of checking.
This article explains, in order, how it happens, where it is likely, four countermeasures, and how to read benchmark accuracy.
The mechanism of predicting the next word produces errors
An LLM reads a large amount of text and learns which words tend to follow which. Sentence forms and phrasing appear many times, so it learns them correctly. A fact that appears only once or twice, such as a particular person's birthday, gives few clues for telling a right answer from a wrong one.
A 2025 paper argues that when true and false statements cannot be told apart, errors are statistically unavoidable. Moreover, many evaluations give points for a correct answer and zero for answering "I don't know". Under this scoring, guessing raises the score, so models are trained to answer even when they do not know.
In other words, plausible errors arise naturally from the way models learn and are evaluated, rather than from a flaw in their construction. Assume that a confident tone says nothing about whether an answer is right.
Where is it likely to happen?
Errors are likely in the parts where correctness cannot be judged from the form of the sentence and the model can answer only if it remembers the fact itself.
- Proper nouns: names of people, companies, products and places
- Numbers: dates, amounts, percentages and statistics
- Citations: titles of books and papers, articles of laws, URLs
- New events: things after the training data cutoff
- Fine points of specialist subjects: knowledge that appears only a little in text for general readers
Citations are especially risky. A model may produce the title of a paper that does not exist, or the number of a legal article that does not exist, in a convincing form. On the other hand, in work that does not depend on factual correctness, such as summarizing, paraphrasing or suggesting a structure, errors do less harm. Work out in advance which parts of a request depend on facts.
Four countermeasures to reduce it
There are four countermeasures. The more you combine, the fewer the errors, but they never reach zero.
- Ask for sources: have the model show the source or quote behind each claim, and withdraw claims for which no support is found
- Pass documents: give the documents the answer should be based on, and instruct the model to answer using only that material. A system that searches for material and passes it in is called RAG
- Let it say it does not know: write in the instructions, "If something is not in the material, you may answer that you don't know"
- Check by hand: open the original material and check proper nouns, numbers and citations
Anthropic also recommends telling the model it may answer that it does not know, and having it show a supporting quote for each claim. Asking the same question several times and looking for parts where the answers disagree also helps find errors.
When asking about internal company rules, for example, paste the rules document and write: "Answer using only this document. If something is not in the document, answer that you don't know. Attach the sentence your answer is based on, quoted exactly from the document." With a quote attached, checking only means opening the quoted passage.
For questions where errors do great harm, such as law, medicine and tax, use the LLM's answer as a starting point for research and confirm in the end with a professional or official material. Anthropic also states that important information should always be checked.
How should you read benchmark accuracy?
Accuracy on a benchmark (a test that measures models on a shared set of problems) is the share of problems answered correctly. Problems answered with "I don't know" and problems answered wrongly are often both counted simply as misses. Under this counting, the more a model guesses, the more easily its accuracy rises.
For work where factual correctness matters, look not only at accuracy but also at the share answered wrongly. Some benchmarks score answers separately. SimpleQA, released in 2024, for example, scores answers to short factual questions in three groups: correct, wrong and not answered. If accuracy is the same, the model with the lower share of wrong answers is the one that takes less checking.
The ratings on the use pages do not directly measure how many untrue answers a model gives. To check on your own work, prepare about 10 questions whose answers you know, and count the errors and the "I don't know" answers separately.
Summary - check only the parts that depend on facts
Hallucination comes from the mechanism of predicting the next word and from evaluations that reward guessing. Since it cannot be eliminated, identify the parts where it is likely, such as proper nouns, numbers, citations and new events, and check them. Before using an answer, mark the parts that depend on facts and check only those against the original material; this takes less work than researching the whole text again.
The countermeasures are four: ask for sources, pass documents, let the model say it does not know, and check by hand. If you often have models answer from documents you pass in, check the ratings for reading long documents on the page for answering from documents.