Home

Published

-

LLM Limits and Hallucinations: Factuality, Evaluation, and QA

An abstract image showing LLM evaluation and quality assurance

Photo by Stephen Dawson on Unsplash

LLM Limits and Hallucinations: Factuality, Evaluation, and QA

1. Executive Summary

LLMs generate plausible text, but they are not truth checkers. Training optimizes next-token prediction, and evaluation often rewards guessing, so uncertainty can still produce confident-looking answers. 出典: Why Language Models Hallucinate and Calibrated Language Models Must Hallucinate support this framing.

Hallucination is therefore not just a bug. It is a predictable result of learning objectives, scoring rules, and decoding behavior that can make guessing attractive. 出典: Why Language Models Hallucinate argues that training and evaluation push models toward guessing, and Calibrated Language Models Must Hallucinate shows that a lower bound remains for arbitrary facts.

The practical question is not whether hallucination exists. The practical question is how much uncertainty the system should tolerate, when it should abstain, and what external checks it needs. 出典: Holistic Evaluation of Language Models argues for multiple evaluation axes, and NIST AI RMF treats evaluation as ongoing risk management.

Benchmarks are useful because they standardize comparison, but benchmark scores do not equal field performance. Public and static datasets can be contaminated, and operational metrics such as latency, cost, refusal rate, and auditability also matter. 出典: An Open Source Data Contamination Report for Large Language Models and The Vulnerability of Language Model Benchmarks: Do They Accurately Reflect True LLM Performance? show why benchmark readings must be cautious.

RAG, citations, tools, and verification help, but none of them guarantee truth. Unsupported claims still appear, citations can fail to reflect the real basis for an answer, and external retrieval opens a path for injection and poisoning. 出典: RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models, Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting, and Indirect Prompt Injection in the Wild: An Empirical Study of Prevalence, Techniques, and Objectives support this claim.

The key takeaways are four.

  1. Hallucination is a side effect of probabilistic generation and depends heavily on training and evaluation design.
  2. Benchmarks are useful, but contamination, overfitting, and distribution shift must always be suspected.
  3. RAG and citations improve grounding, but they do not automatically guarantee fidelity or safety.
  4. Production systems need quality assurance that combines refusal, verification, human review, and audit logs.
   flowchart TD
  A["Training objective"] --> B["Guessing is rewarded"]
  C["Evaluation pressure"] --> B
  B --> D["Uncertain input"]
  D --> E["Plausible generation"]
  E --> F["Hallucination"]
  D --> G["Refuse or defer"]

This diagram treats hallucination as a design outcome rather than a mysterious failure. In practice, the important control point is the ability to stop at D by refusing or verifying.

2. Background and Research History

The starting point of LLM research is Transformer-based autoregressive generation. The model looks at an input sequence and then chooses the next token step by step. Truth checking is not the central operation in that loop. 出典: Attention Is All You Need and Language Models are Few-Shot Learners show this basic structure.

RAG then spread as a way to inject external documents into context. Long context, tool use, benchmark expansion, calibration, and explainability research followed in parallel. 出典: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, Lost in the Middle: How Language Models Use Long Contexts, and Holistic Evaluation of Language Models fit that trajectory.

Hallucination research moved from describing wrong answers to explaining why wrong answers happen systematically. By 2023, the key themes were training-data extraction, position effects in long context, and the limits of calibration and refusal. From 2024 to 2026, RAG fidelity, benchmark contamination, injection attacks, and unfaithful explanations became clearer. 出典: Extracting Training Data from Large Language Models, Scalable Extraction of Training Data from (Production) Language Models, RAGTruth, and An Open Source Data Contamination Report for Large Language Models support this summary.

The lesson from that history is that LLM limits are not confined to one layer. Pretraining, inference, retrieval, evaluation, and auditing can all fail, so a single fix is not enough. 出典: Why Language Models Hallucinate and Holistic Evaluation of Language Models both point toward system-level analysis.

3. Why Hallucinations Happen

The first point is that LLMs are not optimized to tell the truth. In many training setups, the model learns to predict the next token, and evaluation mainly rewards the right answer. That means guessing can be more attractive than saying “I do not know.” 出典: Why Language Models Hallucinate explains the incentive structure from training and evaluation, and Calibrated Language Models Must Hallucinate gives a lower bound for arbitrary facts.

The second point is that decoding is not evidence checking. The model simply emits high-probability continuations step by step, so plausibility and correctness do not always match. Lower temperature does not remove the underlying uncertainty. 出典: Language Models are Few-Shot Learners shows the autoregressive setup, and Calibrated Language Models Must Hallucinate shows that uncertainty remains at inference time.

The third point is that memory and factuality are different things. A model can memorize parts of training data, but it cannot always retrieve them reliably. A fact may be in memory and still be replaced by a plausible alternative when the context shifts. 出典: Extracting Training Data from Large Language Models and Scalable Extraction of Training Data from (Production) Language Models show the gap between memory and retrieval.

The fourth point is that hallucination is not only about low confidence. When the prompt is ambiguous or the context is incomplete, the model may still fill the gap with a plausible completion. That is statistical completion, not semantic verification. 出典: Why Language Models Hallucinate and Holistic Evaluation of Language Models make this distinction easier to see.

The fifth point is that hallucination is not always a fully wrong answer. A sentence can be mostly right while the number, attribute, causal link, or citation target is wrong. In practice, those partial errors are often the most dangerous. 出典: RAGTruth shows that unsupported and contradictory content still appears in RAG systems.

4. Benchmark Evaluation vs Practical Evaluation

A benchmark is a comparison instrument. Practical evaluation is a failure-prevention instrument for a real workflow. They look similar, but they serve different purposes. 出典: Holistic Evaluation of Language Models argues for broader evaluation than a single benchmark score.

Benchmarks are reproducible. But the more public, static, and popular a benchmark becomes, the more likely it is to be affected by overlap with training data or by overfitting. That is why a score alone is not enough. 出典: An Open Source Data Contamination Report for Large Language Models and The Vulnerability of Language Model Benchmarks: Do They Accurately Reflect True LLM Performance? address this problem empirically.

Practical evaluation must be built around the failures that matter in the actual workflow. In legal, medical, financial, support, and internal-search settings, refusal quality, provenance tracking, latency, cost, and stability under reruns can matter more than a plain accuracy number. 出典: NIST AI RMF and Holistic Evaluation of Language Models both imply that accuracy alone is not enough.

DimensionBenchmark evaluationPractical evaluation
GoalCompare modelsAvoid failures in your workflow
DataPublic and fixedInternal docs, logs, hidden cases
WeaknessContamination, overfitting, stalenessDistribution shift, operational cost, permissions
Metricsaccuracy, calibration, robustnesscorrectness, refusal rate, provenance rate, latency, cost
Change rateLowHigh
Failure visibilityOften hidden on a leaderboardExposed in real use

The point of the table is not to reject benchmarks. The point is that benchmarks are necessary but not sufficient. 出典: Holistic Evaluation of Language Models supports a multi-metric design, and contamination research warns against reading static scores too literally.

5. What Helps and What Remains

RAG inserts external documents into context. That improves freshness and makes internal knowledge easier to use. But if the retrieved material is wrong, the answer can still be wrong. 出典: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks gives the basic RAG design, and RAGTruth shows that hallucination can remain even with evidence present.

Citations help expose provenance. But a citation attached by the model is not the same thing as the real basis for the answer. Citations show provenance, but they do not prove fidelity. 出典: Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting shows that a plausible explanation may not reflect the actual reasoning path.

Tool use can add factual checks through calculation, search, database lookups, and external APIs. That is useful, but it also requires permissions, auditing, rollback, and protection against polluted inputs. The convenience of external retrieval creates a new attack surface. 出典: Indirect Prompt Injection in the Wild: An Empirical Study of Prevalence, Techniques, and Objectives and Backdoored Retrievers for Prompt Injection Attacks on Retrieval Augmented Generation of Large Language Models demonstrate that attack surface.

Long context is not a universal fix either. When the relevant information sits in the middle of a long input, the model can still miss it. It is better to decide what should be retrieved, what should be structured, and what should be refused before you simply make the input longer. 出典: Lost in the Middle: How Language Models Use Long Contexts shows the positional limitation.

MethodWhat it improvesWhat remains
RAGFreshness, evidence accessRetrieval errors, contamination, injection
CitationsTraceabilityNot a guarantee of fidelity
Tool useFact checking, calculation, lookupPermissions and auditing are required
Long contextMore material to referencePositional weakness and missed details
Calibration and refusalLess overconfident behaviorA lower bound remains for arbitrary facts

The point is not that one method solves everything. The point is that the failure mode you want to reduce usually needs a stack of methods, not one trick. 出典: RAGTruth, Lost in the Middle, and Calibrated Language Models Must Hallucinate make that clear.

6. A Quality Assurance Process

What follows is a public-information inference. In high-risk LLM use, you need quality assurance for the whole system, not only for the model. 出典: NIST AI RMF and Holistic Evaluation of Language Models support this inference.

First, separate questions into facts, reasoning, summaries, calculations, and actions. Facts go to source checks, reasoning goes to condition checks, summaries go to fidelity checks, calculations go to tools, and actions go to permission checks. 出典: This split is a public-information inference from the separate roles of RAG, tools, and evaluation axes.

Next, keep evaluation in three layers. Use public benchmarks to check breadth, a private holdout to check workflow fit, and an adversarial set to check hallucination and injection. 出典: An Open Source Data Contamination Report for Large Language Models and Indirect Prompt Injection in the Wild support why a single fixed set is not enough.

Then measure the following every time.

  1. Factuality.
  2. Citation support.
  3. Refusal quality.
  4. Latency.
  5. Cost.
  6. Stability under reruns.
   flowchart TD
  A["Question classification"] --> B["Gather evidence"]
  B --> C["Automated checks"]
  C --> D["Human review"]
  D --> E["Post-launch monitoring"]
  C --> F["Fix and rerun"]
  F --> C

The point of the flow is not to hope that the model becomes correct. The point is to find failures early, reproduce them, isolate them, and watch for regressions. 出典: NIST AI RMF emphasizes ongoing management, and Holistic Evaluation of Language Models emphasizes multi-dimensional measurement.

7. Risks and Limits

First, zero hallucination is not realistic. For arbitrary facts, there will always be residual error under the structure of training and evaluation. The practical question is not zero hallucination, but the acceptable failure rate. 出典: Calibrated Language Models Must Hallucinate and Why Language Models Hallucinate support that point.

Second, explanations are not proofs. LLMs can produce plausible reasons, but those reasons may not reflect the actual internal process. Explanations should therefore not replace audit evidence. 出典: Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting shows this limitation.

Third, RAG improves freshness, but it is not a truth guarantee. Retrieval sources can be contaminated, and injection attacks can slip in instructions or malicious documents that should not have been there. 出典: RAGTruth and Indirect Prompt Injection in the Wild show the residual risk.

Fourth, benchmarks are fragile. Public data is easy to contaminate, and a number does not map cleanly to a workflow failure mode. A high score is not the same as practical usefulness. 出典: An Open Source Data Contamination Report for Large Language Models and The Vulnerability of Language Model Benchmarks: Do They Accurately Reflect True LLM Performance? are the main warnings here.

8. Recommendations

  1. Separate questions the model can answer from questions it should refuse.
  2. Decide on refusal and deferral behavior before deployment.
  3. Externalize evidence through RAG and tool use.
  4. Keep citations, but do not treat citations themselves as proof.
  5. Use benchmarks, but supplement them with internal holdouts and adversarial sets.
  6. For high-risk use cases, retain human review and audit logs.

This sequence is sensible because finding and stopping failures early is often more effective than trying to make the model smarter. 出典: NIST AI RMF, Holistic Evaluation of Language Models, and RAGTruth together support this public-information inference.

References