Reducing LLM hallucinations in production systems

You've shipped an agent. It works most of the time. Then a user reports that it confidently invented a price, made up a customer record, or claimed a feature exists when it doesn't. The model hallucinated, and your system passed it through.

Hallucinations aren't a solvable problem—they're a property of how LLMs work. But you can reduce their surface area and catch them before they reach users. This is architecture work, not prompt engineering.

The core issue

LLMs generate text token by token based on probability. They have no ground truth, no real-time access to your systems, and no inherent way to distinguish between plausible-sounding text and correct text. A well-trained model is very good at producing fluent, coherent hallucinations.

Reducing hallucinations means: (1) giving the model less opportunity to invent, and (2) catching what it does invent before it matters.

Grounding: The first line of defense

The most effective single change is to stop asking the model to know things. Instead, retrieve or compute the answer first, then ask the model to work with concrete data.

If you need customer information, fetch it. If you need current prices, query them. If you need to know whether a feature exists, check your feature flag system. Then put that data in context and ask the model to use it.

User: "What's the shipping cost to Chicago?"
  ↓
[Retrieve: shipping rules, customer location, current rates]
  ↓
System prompt: "Given the shipping rules below, calculate cost"
  ↓
Model: "Based on the provided rules, the cost is $12.50"

This is tool use, retrieval-augmented generation (RAG), or just good system design. The model becomes a processor of concrete data rather than a generator of plausible text.

The tradeoff: You need to build the retrieval layer. You need to keep it fast and accurate. You need to handle cases where the data is incomplete or stale. But this is solvable infrastructure work, not a language model problem.

Verification layers

Even with grounding, the model can misinterpret data or make logical errors. Add a verification step before execution.

For structured outputs (a booking, a price calculation, a decision), parse the response and check it against rules:

• Does the calculated price match the formula given?
• Is the recommended action valid for this state?
• Are all required fields present and in valid ranges?

If verification fails, catch it. Log it. Either retry with a corrected prompt, or surface it as "I couldn't complete this" rather than returning a wrong answer.

def execute_with_verification(prompt, verify_fn):
    response = model.generate(prompt)
    parsed = parse_response(response)
    
    if not verify_fn(parsed):
        logger.warning("Verification failed", response=response)
        return None  # Don't pass through
    
    return parsed

This catches a category of errors: logical inconsistencies, constraint violations, impossible states. It won't catch all hallucinations (a model can invent data that passes your checks), but it will catch the ones that break your system.

The tradeoff: You need to define what "correct" means for each action. This is extra code. But it's deterministic and testable, unlike the model itself.

Confidence and uncertainty

Some models expose a confidence score or can be prompted to express uncertainty. Use this.

If the model says "I'm not sure" or returns a low confidence score, treat it as a signal to either (1) retrieve more data, (2) ask a follow-up question, or (3) escalate to a human.

If confidence < 0.7:
  → Retrieve additional context
  → If still uncertain, ask user for clarification
  → If still uncertain, escalate

The tradeoff: Confidence scores aren't reliable. A model can be confidently wrong. But they're a useful signal in combination with other checks. Don't rely on them alone.

Routing and fallbacks

Not all tasks should go to an LLM. Some should go to a rule engine or a deterministic system.

"What's the balance on account 12345?" should not be an LLM call. It should be a database lookup. "Summarize this support ticket" might be an LLM call, with fallback to "unable to summarize."

flowchart TD
    A["User request"] --> B{Can answer<br/>deterministically?}
    B -->|Yes| C["Query system<br/>of record"]
    B -->|No| D{Requires<br/>external data?}
    D -->|Yes| E["Retrieve data<br/>first"]
    D -->|No| F["Call LLM<br/>with context"]
    C --> G["Return answer"]
    E --> F
    F --> H{Verify<br/>output}
    H -->|Pass| G
    H -->|Fail| I["Escalate or<br/>retry"]

The tradeoff: You're adding routing logic. But you're also reducing the number of hallucinations that matter. A hallucination in a summary is less critical than a hallucination in a transaction amount.

Evals and monitoring

You can't fix what you don't see. Build evals that catch hallucinations in your specific domain.

• Fact-checking: Does the model claim something that contradicts your data?
• Consistency: Does the model contradict itself across multiple calls?
• Constraint violations: Does the output break rules you've defined?

Run these evals on a sample of production traffic. Log failures. Treat hallucinations as a metric, like error rate.

def eval_hallucination(response, ground_truth):
    # Did the model invent facts?
    facts_claimed = extract_facts(response)
    facts_true = extract_facts(ground_truth)
    
    invented = facts_claimed - facts_true
    return len(invented) == 0

The tradeoff: You need domain knowledge to write good evals. But without them, you're flying blind. You'll ship hallucinations without knowing it.

What doesn't work

Prompt engineering alone won't solve this. Telling the model "be accurate" or "double-check your work" helps a little, but the model still has no ground truth. It's like asking someone to verify a fact they don't have access to.

Fine-tuning on hallucination-free data helps, but only for patterns the training data covered. New domains, new products, new rules—the model will still invent.

The realistic approach

In production, you're probably doing all of this:

1. Ground everything you can. Fetch data before asking the model to reason about it.
2. Verify outputs. Parse, check constraints, fail safely.
3. Route intelligently. Don't use an LLM where a lookup or rule engine works.
4. Monitor and eval. Know your hallucination rate by category.
5. Escalate when uncertain. If verification fails, don't guess. Ask a human or retry with better data.

This isn't elegant. It's not a one-line fix. But it's how you ship systems that don't break when the model confabulates.

The model is a component in a larger system. The system is responsible for correctness, not the model. Design accordingly.