Handling LLM rate limits and retries in production

You're in production. Your agent hits Claude or GPT-4 and gets a 429. You retry. Everyone retries. The API is now more overloaded. Your retry made things worse.

This is the core problem with rate limit handling: naive retry logic is a denial-of-service attack on the service you depend on. You need to think about this as a system, not just error handling.

The failure modes

Rate limits come in several flavors, and they matter:

Soft limits are per-minute or per-second quotas. You hit them when traffic spikes. The API is fine; you just need to back off. These are the ones you can safely retry.

Hard limits are account-level caps. You've genuinely exceeded your allocation. Retrying immediately is pointless. You need to either wait until the window resets or fail fast and queue the work.

Cascading failures happen when your retry logic is synchronous. One slow request blocks your worker. More requests pile up. You hit the rate limit harder. Now everything is slow. This is where most teams go wrong.

The other failure mode is retry amplification. If you have ten concurrent requests and each one retries three times on a 429, you've just sent thirty requests at an already-overloaded API. If they all retry again, you're at ninety. This compounds fast.

What actually works

The shape of a production solution looks like this:

1. Classify the error. Is this a 429? A 500? A timeout? Each needs different handling.
2. Use exponential backoff with jitter for transient errors. Not linear backoff. Not fixed intervals. Jitter prevents thundering herds.
3. Respect Retry-After headers if the API provides them. Claude and OpenAI both do. Use the value they tell you.
4. Fail fast for hard limits. If you've hit your monthly quota, retrying won't help. Queue the work or reject it.
5. Decouple retries from the request path. Don't retry synchronously in your handler. Move the work to a queue.

Here's what the decision tree looks like:

flowchart TD
    A["LLM API call"] --> B{Response status}
    B -->|429| C{Retry-After header?}
    B -->|5xx| D[Exponential backoff<br/>max 3 retries]
    B -->|4xx other| E[Fail immediately]
    B -->|2xx| F[Success]
    C -->|Yes| G[Wait specified time<br/>then retry]
    C -->|No| H[Exponential backoff<br/>max 2 retries]
    G --> I{Retry succeeded?}
    H --> I
    I -->|No| J[Send to retry queue]
    I -->|Yes| F
    D --> K{Retry succeeded?}
    K -->|No| L[Log and fail]
    K -->|Yes| F

The code shape

Here's a minimal example for a single request:

import random
import time
from typing import Optional

def call_with_retry(
    api_call,
    max_retries: int = 2,
    base_delay: float = 1.0,
) -> Optional[dict]:
    """Call LLM API with exponential backoff and jitter."""
    
    for attempt in range(max_retries + 1):
        try:
            return api_call()
        except RateLimitError as e:
            retry_after = e.retry_after  # Seconds from API
            if attempt == max_retries:
                raise
            
            # Use provided Retry-After or compute backoff
            if retry_after:
                wait_time = retry_after
            else:
                # Exponential backoff: 1s, 2s, 4s + jitter
                wait_time = base_delay * (2 ** attempt)
                jitter = random.uniform(0, wait_time * 0.1)
                wait_time += jitter
            
            time.sleep(wait_time)
        except TransientError:
            if attempt == max_retries:
                raise
            wait_time = base_delay * (2 ** attempt)
            jitter = random.uniform(0, wait_time * 0.1)
            time.sleep(wait_time + jitter)
    
    return None

This works for a single request. But in production, you have many requests. Sleeping in your request handler is a mistake. You're blocking a worker.

Decoupling retries from the request path

The real pattern is to use a task queue:

# In your request handler - fail fast
def handle_agent_request(user_id: str, prompt: str):
    try:
        result = call_llm_with_retry(prompt, max_retries=1)
        return {"status": "success", "result": result}
    except RateLimitError:
        # Queue for retry, don't block the request
        task_queue.enqueue(
            "retry_llm_call",
            user_id=user_id,
            prompt=prompt,
            retry_count=1,
        )
        return {"status": "queued"}
    except Exception as e:
        return {"status": "error", "message": str(e)}

# In your background worker - retry with patience
def retry_llm_call(user_id: str, prompt: str, retry_count: int):
    max_retries = 5
    if retry_count > max_retries:
        log_failure(user_id, prompt, "exceeded max retries")
        return
    
    try:
        result = call_llm_with_retry(prompt, max_retries=1)
        notify_user(user_id, result)
    except RateLimitError:
        # Exponential backoff between queue attempts
        delay = 2 ** retry_count  # 2, 4, 8, 16, 32 seconds
        task_queue.enqueue_at(
            "retry_llm_call",
            delay_seconds=delay,
            user_id=user_id,
            prompt=prompt,
            retry_count=retry_count + 1,
        )
    except Exception as e:
        log_failure(user_id, prompt, str(e))

The key insight: retrying in the request handler is for transient blips. Retrying in a background worker is for sustained rate limiting. They have different retry budgets and different backoff strategies.

Monitoring and tuning

You need visibility into what's actually happening:

• Track the ratio of successful requests to 429s. If it's climbing, your quota is too tight or your traffic is growing.
• Log Retry-After values. If the API is consistently asking you to wait 30+ seconds, you're hitting hard limits.
• Measure queue depth. If tasks are piling up in the retry queue, you need to either reduce traffic or increase your quota.
• Set alerts on retry exhaustion. If tasks are hitting max retries consistently, something is wrong.

One practical tuning knob: adjust max_retries based on whether you're in the request path or the background worker. In the request path, 1-2 retries. In the background worker, 5-10 retries with longer backoff.

Tradeoffs to know

Exponential backoff with jitter is not magic. It helps spread load, but if you're genuinely over quota, you're over quota. Backoff just delays the inevitable. You need to either reduce traffic or increase your quota.

Queuing adds latency. If you queue everything on a 429, users won't get results immediately. This is often the right tradeoff—better to be slow than to fail—but it's a tradeoff.

Retry-After headers are advisory. Some APIs send them, some don't. Some send nonsensical values. Respect them when they exist, but have a sensible default (exponential backoff).

Jitter matters more than you think. Without it, all your retries happen at the same time. With it, they spread out. This is the difference between a retry storm and a controlled recovery.

The honest answer: there's no perfect solution. Rate limits exist to protect the API. Your job is to respect them while keeping your system reliable. Exponential backoff, jitter, queuing, and monitoring get you most of the way there. The rest is tuning based on your specific traffic patterns and quota.