LLM API Error Handling: Retries, Rate Limits, Timeouts, and Fallbacks

Overview

LLM API failures usually fall into a small number of operational categories: temporary provider errors, client or authentication errors, rate limits, network failures, slow responses, and responses that arrive successfully but cannot be used safely. Treating every failure as a generic exception makes recovery unpredictable. A production integration should classify the event first, then apply a response appropriate to that class.

The basic workflow is:

1. Set a deadline for the complete request and, where supported, separate connection, read, and overall timeouts.
2. Capture the provider response, status code, request identifier, model, latency, and a redacted error context.
3. Classify the failure as retryable, non-retryable, or uncertain.
4. Retry only within a bounded budget and use exponential backoff with jitter.
5. Validate the response before passing it to downstream code.
6. Use a fallback model, provider, cached result, or human review path when the primary route cannot complete.

This design is useful whether your application performs chat, extraction, summarization, classification, retrieval-augmented generation, or structured data generation. For response validation, see this guide to structured output prompts. Error handling should complement prompt design rather than compensate for an unclear prompt or an underspecified output contract.

What to track

1. Classify the error response

Provider terminology and status behavior can differ, so confirm the details against the SDK and API documentation you use. The following reference is a provider-neutral starting point rather than a universal mapping.

Do not log prompts, documents, credentials, or generated content indiscriminately. Establish a redaction policy and retain enough metadata to correlate an incident without exposing sensitive data. Useful metrics include success rate, retry rate, timeout rate, rate-limit events, fallback rate, validation failures, latency by attempt, and total request cost where your platform exposes a reliable usage measure.

2. Track the retry budget

A retry policy needs explicit limits. Define a maximum number of attempts, a maximum total elapsed time, and—if relevant—a maximum retry delay. A request that can wait several seconds in an internal batch job may be unsuitable for an interactive endpoint. Keep the retry budget separate from the user-facing request timeout so the service cannot spend the entire deadline sleeping between attempts.

3. Identify whether an operation is safe to repeat

Most LLM requests are logically read-like: repeating them does not directly create a duplicate external action. However, an LLM response may trigger an email, payment, database update, ticket, or workflow step. A network failure after the model responded leaves the client uncertain about completion. Use an idempotency key or durable operation record for any workflow where repeating the request could create a duplicate effect.

An idempotency key should represent the logical operation, not merely a transport attempt. Store its status, input fingerprint, provider request identifiers, and final result when appropriate. On a repeated request, return the recorded result or resume the operation instead of starting an uncontrolled duplicate.

Cadence and checkpoints

Reliability settings should be reviewed on a regular schedule and whenever the integration changes. A monthly operational review is useful for active systems; a quarterly review is a reasonable minimum for stable systems. The correct cadence depends on request volume, business impact, provider changes, and how frequently the application is deployed.

Step 1: Establish a baseline

Before changing retry settings, record a baseline for successful requests, failures by category, p50 and high-percentile latency, timeout frequency, rate-limit events, fallback usage, and malformed responses. Segment the data by model, provider, endpoint, request type, and application version. A single aggregate success rate can conceal a problem affecting one important workflow.

Step 2: Verify timeout layers

Map every timeout in the request path: client, application server, reverse proxy, queue, provider SDK, and database or tool calls invoked after the model response. The outer deadline should leave enough time to return a controlled result. Streaming requires separate thought: decide how long to wait for the first token, how long a gap between tokens is acceptable, and what to do if a stream stops halfway through a structured response.

Step 3: Review rate-limit behavior

Check the provider's current documentation and SDK behavior rather than assuming that all limits are expressed the same way. Limits may concern requests, tokens, concurrency, accounts, models, or a combination of these. Confirm whether the response includes a retry hint, how the SDK exposes it, and whether your queue or worker system can apply backpressure.

Step 4: Exercise failure paths

Use controlled tests to simulate timeouts, connection resets, retryable server errors, rate limits, invalid responses, and provider unavailability. Verify that the system does not retry authentication failures indefinitely, does not send duplicate side effects, and does not expose raw provider errors to users. Include tests for cancellation: if a user abandons a request, the server should stop unnecessary work where the platform permits it.

Step 5: Recheck dependencies

When upgrading an SDK or adding a provider, inspect exception types, timeout defaults, streaming behavior, response schemas, retry defaults, and request identifiers. An SDK may add its own automatic retries. If your application also retries, the effective attempt count may be higher than expected. Make one layer the clear owner of retries or coordinate the policies explicitly.

How to interpret changes

Reliability metrics are most useful when read together. A rising retry rate with stable success may indicate temporary provider instability, but it can also mean the application is retrying requests that should be corrected. A rising latency curve with fewer hard errors may show that retries are masking an outage while increasing user wait time. A growing fallback rate may indicate a primary model problem, overly aggressive thresholds, or a fallback that is not meeting quality requirements.

Use these interpretations as investigation prompts:

• More rate limits: inspect concurrency, token volume, queue behavior, and traffic spikes before simply increasing retries.
• More timeouts: compare prompt size, requested output length, model selection, network latency, and downstream deadlines.
• More malformed responses: inspect schema validation, truncation, prompt changes, model changes, and whether the parser accepts only the documented format.
• More fallback calls: compare fallback quality, latency, and cost with the primary route; a fallback should be tested, not merely configured.
• More duplicate operations: review idempotency persistence, worker acknowledgements, and the point at which an external side effect is committed.

Exponential backoff and jitter

Exponential backoff increases the delay between attempts while jitter prevents many workers from retrying simultaneously. Keep the delay bounded and honor a provider-supplied retry interval when it is available and valid.

function callWithRetry(operation, deadline, maxAttempts):
    start = clock.now()

    for attempt in 1..maxAttempts:
        remaining = deadline - clock.now()
        if remaining <= 0:
            return timeout_failure()

        try:
            return operation(timeout=remaining)
        catch error:
            classification = classify(error)
            if classification == NON_RETRYABLE:
                return fail(error)
            if classification == UNCERTAIN_SIDE_EFFECT:
                return reconcile_or_fail(error)
            if attempt == maxAttempts:
                return fail(error)

            baseDelay = min(MAX_DELAY, INITIAL_DELAY * 2^(attempt - 1))
            providerDelay = valid_retry_hint(error)
            delay = max(baseDelay, providerDelay) if providerDelay exists else baseDelay
            sleep(random_between(0, delay))

    return fail("retry budget exhausted")

The exact values for initial delay, maximum delay, attempts, and deadline belong in configuration, not scattered through application code. Tune them using observed latency and failure patterns. More retries are not automatically more reliable if they overload a struggling provider or cause requests to miss their user-facing deadline.

Idempotency pattern

function process_request(operationKey, input):
    fingerprint = hash(normalize(input))
    record = store.lookup(operationKey)

    if record exists:
        if record.fingerprint != fingerprint:
            return conflict("key reused for different input")
        if record.status == COMPLETE:
            return record.result
        if record.status == IN_PROGRESS:
            return wait_or_poll(record)

    store.create_if_absent(
        key=operationKey,
        fingerprint=fingerprint,
        status=IN_PROGRESS
    )

    result = run_model_request_with_deadline(input)
    validated = validate_schema(result)
    store.complete(operationKey, validated)
    return validated

For distributed systems, make the record creation atomic and define recovery for abandoned IN_PROGRESS records. If the model call or external action may have completed before the worker failed, reconcile before restarting.

When to revisit

Revisit this article and your implementation whenever you add a model or provider, change an SDK, alter prompt or output schemas, introduce streaming, increase traffic, add tool calls, or move from synchronous requests to queues. Also review it after a provider incident, a recurring timeout pattern, a new compliance requirement, or any duplicate side effect.

For a practical quarterly checkpoint, export the previous period's metrics and answer five questions:

1. Which error categories increased, and which workflows were affected?
2. Did retries improve completed requests without creating unacceptable latency or load?
3. Are timeouts aligned across the client, service, queue, and provider SDK?
4. Did malformed responses or schema failures change after prompt, model, or SDK updates?
5. Can the fallback path deliver an acceptable result, or should it route to a queue or human review?

Record the answers with the relevant model versions, SDK versions, configuration changes, and test results. Link incidents to the dashboards and runbooks used by the team. Related implementation work may also benefit from a prompt testing framework, a RAG reliability review, or guidance on human-in-the-loop AI workflow automation.

Start with one endpoint: add error classification, bounded retries, deadlines, response validation, and metrics. Test each failure path deliberately, then extend the same policy to other LLM integrations. This incremental approach makes recurring failures visible and gives you a dependable baseline to update as the application evolves.