How to evaluate LLM outputs in production

You shipped an agent. It works in your notebook. Then in production it hallucinates a customer's account number, or returns a response in the wrong format, or gets stuck in a loop. You need to know this happened before the customer does.

Evaluating LLM outputs is not like testing traditional software. You can't assert that the output equals some hardcoded expected value. The output space is infinite. But you also can't just ship it and hope. You need signals that tell you when things are breaking.

The shape of the problem

Start by naming what you're actually evaluating. Are you checking that:

• The response answers the question asked
• The tool calls are syntactically valid and semantically sensible
• The reasoning is grounded in retrieved facts
• The output format matches what downstream systems expect
• The response doesn't violate safety or policy constraints

These are different problems. You need different signals for each.

The honest version: you will combine multiple evaluation methods. No single approach catches everything. You layer them. Some run synchronously before returning to the user. Some run asynchronously on production traffic. Some are automated. Some involve human review.

Synchronous checks: catch it before you ship it

These run in your request path. They need to be fast. They block the response.

Format validation. If your LLM is supposed to return JSON, validate the JSON. Parse it. Check required fields exist. This is free—do it always.

import json

def validate_response_format(response_text, schema):
    try:
        data = json.loads(response_text)
    except json.JSONDecodeError:
        return False, "Invalid JSON"
    
    required = schema.get("required", [])
    for field in required:
        if field not in data:
            return False, f"Missing field: {field}"
    
    return True, None

Tool call validation. If your agent uses tools, validate that the tool calls are real tools your system knows about, with arguments in the right shape. Don't let the LLM invent tools.

def validate_tool_calls(calls, available_tools):
    for call in calls:
        if call["tool_name"] not in available_tools:
            return False, f"Unknown tool: {call['tool_name']}"
        
        tool_schema = available_tools[call["tool_name"]]
        required_args = tool_schema.get("required_args", [])
        for arg in required_args:
            if arg not in call.get("arguments", {}):
                return False, f"Missing arg {arg} for {call['tool_name']}"
    
    return True, None

Guardrails. Check for obvious policy violations. Does the response contain PII that shouldn't be there. Does it refuse when it shouldn't, or comply when it absolutely shouldn't. These are domain-specific, but they're binary checks you can run fast.

If validation fails, you have choices: return an error to the user, retry with a different prompt, or fall back to a safer response. What you do depends on your SLA and the cost of getting it wrong.

Scoring: the middle ground

You can't check everything synchronously. You need to score outputs on dimensions that matter and act on the scores.

The pattern: for each dimension, write a scorer that takes the LLM output (and usually the input and context) and returns a score, usually 0-1. Aggregate the scores. Threshold them. Route based on the result.

def score_relevance(response, query, retrieved_docs):
    """Does response draw from retrieved context?"""
    # Simple: check if response mentions key entities from docs
    doc_entities = extract_entities(retrieved_docs)
    response_entities = extract_entities(response)
    overlap = len(doc_entities & response_entities) / len(doc_entities)
    return overlap

def score_completeness(response, query):
    """Does response address all parts of the query?"""
    # Simple: check if response is longer than minimum
    # Better: check if it addresses each question in query
    return len(response.split()) > 50  # stub

def aggregate_score(response, query, context):
    relevance = score_relevance(response, query, context)
    completeness = score_completeness(response, query)
    return 0.7 * relevance + 0.3 * completeness

The hard part is not writing scorers. It's deciding what to score and what threshold means "reject this."

Start with scorers that are deterministic or use lightweight models. Regex-based checks. Keyword matching. Length checks. If you need semantic understanding, use a small model or a cheaper LLM call, not your main inference path.

The threshold is a business decision. If you set it too high, you reject good outputs and degrade user experience. Too low, and bad outputs reach users. You find it by running your eval on production traffic (logged, not live) and measuring what threshold would have caught real failures.

def route_response(score, response):
    if score > 0.85:
        return ("ship", response)
    elif score > 0.60:
        return ("human_review", response)
    else:
        return ("reject", response)

Asynchronous evaluation: learn from production

What you don't catch synchronously, you catch after the fact. This is where you get signal about systemic failures.

Log everything. The input, the output, the retrieval context, the tool calls, the timestamp. This is your ground truth for debugging.

Run evals on a sample of production traffic. You can be more expensive here—use a better model, use human review for a subset, whatever you need. The goal is to find patterns in what's failing.

# Pseudo-code: daily eval job
def evaluate_production_sample(logs, sample_size=500):
    sample = random.sample(logs, min(sample_size, len(logs)))
    
    failures = []
    for log in sample:
        # Expensive eval: call Claude to judge correctness
        is_correct = eval_with_gpt4(log["input"], log["output"])
        if not is_correct:
            failures.append(log)
    
    # Analyze patterns
    print(f"Failure rate: {len(failures) / len(sample)}")
    print(f"Common failure mode: {most_common_error_type(failures)}")
    
    # Alert if above threshold
    if len(failures) / len(sample) > 0.1:
        alert("Eval failure rate above 10%")

The evaluation loop

flowchart TD
    A["LLM generates output"] --> B["Sync validation"]
    B -->|Invalid| C["Reject or retry"]
    B -->|Valid| D["Score output"]
    D -->|High score| E["Ship to user"]
    D -->|Medium score| F["Human review queue"]
    D -->|Low score| G["Reject or fallback"]
    E --> H["Log output"]
    F --> H
    G --> H
    H --> I["Daily async eval"]
    I -->|Failures detected| J["Alert / investigate"]
    I -->|Patterns found| K["Update eval thresholds"]
    K --> B

The loop: sync validation catches obvious failures. Scoring routes uncertain outputs. Async eval finds systematic issues. You use what you learn to tighten sync checks and adjust thresholds.

Honest tradeoffs

You can't evaluate everything. Every eval you add has latency cost. Every threshold you set will let some bad outputs through and reject some good ones.

The LLM you use for evaluation matters. If you use GPT-4 to evaluate your own outputs, you're benchmarking against a different model's capabilities. If you use a cheaper model, you get faster eval but lower accuracy. Most teams use multiple models—fast cheap ones synchronously, expensive accurate ones asynchronously.

Human evaluation doesn't scale. You can't manually review every output. Use it for:

• Calibrating automated evals (run human + automated on same sample, measure correlation)
• Spot-checking failure modes
• Building ground truth for fine-tuning a classifier if you need higher volume

The gap between eval and reality stays real. An output can pass all your evals and still fail in production because you didn't think to check for that failure mode. This is why you log everything and keep reviewing.

What you actually do

Pick the three most important failure modes for your system. Write scorers or checks for those. Set thresholds based on a sample of real traffic. Log everything. Run weekly evals on production samples. Adjust thresholds when you see patterns. Add new checks when you find new failure modes.

It's not elegant. It's not complete. But it's what catches things before they become incidents.