Handle LLM Rate Limits and Retries

By Kokil Thapa | Last reviewed: September 2026

Your app calls an LLM on every form submit, chat message, or document scan. The first hundred requests feel fine. Then traffic spikes, batch jobs overlap, and OpenAI or Anthropic returns HTTP 429. Users see timeouts. Costs spike from blind retries. To handle LLM rate limits and retries properly, you need more than a sleep loop in a controller. You need provider-aware throttling, structured backoff, and queue discipline baked into your architecture from day one.

What HTTP status codes and headers signal LLM rate limits?

Most hosted LLM APIs return 429 Too Many Requests when you exceed a quota. Some providers also return 503 Service Unavailable during overload. Treat both as transient for retry logic, but log them differently. A 429 usually means your account hit RPM, TPM, or RPD caps. A 503 often means regional capacity is strained.

Read the response body. OpenAI-style payloads include an error.type field such as rate_limit_exceeded or insufficient_quota. The second case is not a retry problem. Retrying a billing or quota failure wastes money and delays the real fix.

Check headers before you guess a backoff interval. The Retry-After header tells you how long to wait. It may be seconds or an HTTP date. Honor it when present. Fall back to exponential backoff only when the header is missing.

Provider docs define the limits you are actually fighting. Review the current OpenAI rate limits guide and your vendor tier before you tune workers. Limits differ by model, endpoint, and account tier. Embedding endpoints use token-per-minute budgets. Chat completions use both RPM and TPM.

Common limit types you will hit

• RPM — requests per minute. Batch importers blow this first.
• TPM — tokens per minute. Long prompts plus big completions burn TPM fast.
• RPD — requests per day on free or low tiers.
• Concurrent requests — parallel workers count even when RPM looks fine.

Log limit type, model name, and approximate token count on every 429. That data drives capacity planning. It also feeds AI rate limits and cost optimization decisions later.

How do you implement exponential backoff for LLM API calls?

Naive retry loops cause retry storms. Ten workers each retry five times and you suddenly send fifty calls into a provider that already told you to slow down. Exponential backoff increases wait time after each failure. Jitter randomizes the delay so workers do not wake up together.

A practical formula: base delay × 2^attempt, capped at 60 seconds, plus random jitter up to 25% of the delay. Respect Retry-After when it exceeds your calculated value. Set a max attempt count. Three to five retries is typical for chat. Batch jobs can afford more because they run async.

In PHP 8.3+ on Laravel 12 or Laravel 13, wrap the HTTP call in a small service class. Keep retry policy out of Blade views and fat controllers. I follow the same pattern I use for payment gateways and SMS APIs on production apps.

<?php

namespace App\Services\Llm;

use Illuminate\Http\Client\RequestException;
use Illuminate\Support\Facades\Http;
use RuntimeException;

final class LlmClient
{
    private const MAX_ATTEMPTS = 5;

    public function chat(array $payload): array
    {
        $attempt = 0;

        while (true) {
            $attempt++;

            try {
                $response = Http::withToken(config('services.openai.key'))
                    ->timeout(120)
                    ->post('https://api.openai.com/v1/chat/completions', $payload)
                    ->throw();

                return $response->json();
            } catch (RequestException $e) {
                $response = $e->response;
                $status = $response?->status();

                if ($status === 429 && $attempt < self::MAX_ATTEMPTS) {
                    $this->sleepForRateLimit($response->header('Retry-After'), $attempt);
                    continue;
                }

                if ($status === 503 && $attempt < self::MAX_ATTEMPTS) {
                    $this->sleepForRateLimit(null, $attempt);
                    continue;
                }

                throw $e;
            }
        }
    }

    private function sleepForRateLimit(?string $retryAfter, int $attempt): void
    {
        if ($retryAfter !== null && ctype_digit($retryAfter)) {
            sleep((int) $retryAfter);
            return;
        }

        $baseMs = 500;
        $delayMs = min($baseMs * (2 ** ($attempt - 1)), 60_000);
        $jitterMs = random_int(0, (int) ($delayMs * 0.25));

        usleep(($delayMs + $jitterMs) * 1000);
    }
}

Never block a web request for sixty seconds while waiting on backoff. Return a queued job ID to the browser instead. Users on a legal-tech portal or eCommerce checkout will abandon the page long before the LLM responds. See streaming LLM responses with SSE for a better UX on chat features.

Make retries safe with idempotency keys

LLM calls cost money. A retried chat turn can produce two billable completions for one user message. Store an idempotency key per user action. Check it before you call the provider. Persist the final response keyed to that ID.

This mirrors payment callback handling. On a client portal I built, document summarization runs in a queue with a UUID per upload batch. Retries reuse the same key. The user never sees duplicate summaries. Billing stays predictable.

How should you queue and throttle LLM requests in production?

Retries fix transient failures. Throttling prevents them. Put every non-trivial LLM call on a queue worker. Size concurrency to stay under your TPM budget with headroom. If your tier allows 60 RPM, do not run 60 parallel queue workers each firing one request per second.

Estimate tokens before you send when possible. Count input tokens with the provider tokenizer or a rough character heuristic. Reserve TPM budget in Redis before the HTTP call. Release unused reservation after the response returns. This pattern is simpler than it sounds and beats reactive 429 handling alone.

1. Create a dedicated queue named llm with low concurrency.
2. Track rolling RPM and TPM in Redis with a sliding window.
3. Delay jobs that would exceed budget instead of sending and retrying.
4. Alert when queue depth grows beyond a threshold for fifteen minutes.
5. Route dead-letter failures to a manual review table.

Laravel's rate limiter works well for outbound API pacing. The same concepts apply as in rate limiting and API throttling in Laravel and Laravel rate limiting with custom keys. Use a custom key like llm:openai:tpm rather than per-user keys for provider caps.

<?php

use Illuminate\Support\Facades\RateLimiter;
use Illuminate\Support\Facades\Redis;

final class LlmThrottle
{
    public function acquire(int $estimatedTokens): void
    {
        while (! RateLimiter::attempt(
            'llm-rpm',
            50,
            fn () => true,
            60
        )) {
            usleep(200_000);
        }

        $bucket = 'llm:tpm:' . now()->format('YmdHi');

        while ((int) Redis::get($bucket) + $estimatedTokens > 90000) {
            usleep(500_000);
        }

        Redis::incrby($bucket, $estimatedTokens);
        Redis::expire($bucket, 120);
    }
}

Adjust the TPM ceiling to your actual tier. The number above is illustrative. Pull real limits from your provider dashboard. Pair throttling with monitoring described in LLMOps monitoring and guardrails.

Should you use multiple LLM providers to avoid rate limits?

Multi-provider routing helps when you need availability, not when you want to bypass a single quota you already maxed out. Each provider has its own billing, limits, and output quality profile. Fallback from OpenAI to Anthropic on 429 can keep a chatbot online during a spike. It also doubles your integration and eval surface.

Use a primary and secondary provider with explicit routing rules. Fail over only on 429 or 503, not on validation errors or content policy blocks. Log which provider served each request. Compare quality in LLM output evaluation pipelines before you automate failover in customer-facing paths.

Self-hosted overflow via Ollama or vLLM is viable for internal tools. It is rarely the first fix for a public marketing site. Read running LLMs locally with Ollama and vLLM before you buy GPU hardware. For most Nepal SMB clients I work with, queue discipline plus a paid tier upgrade solves the problem faster than running local inference.

What production mistakes break LLM retry logic?

The failures I see in production are predictable. Teams retry non-idempotent writes. They ignore insufficient_quota. They run LLM calls synchronously inside HTTP requests. They set queue worker counts based on CPU cores instead of provider RPM.

Another common bug: retrying 400-level validation errors. If the prompt is too long, retrying will never succeed. Inspect status and error type before you sleep. Parse JSON error bodies defensively. Providers change field names occasionally.

Security matters too. Do not log full prompts containing PII when debugging 429 storms. Redact tokens and secrets. Follow protecting PII and secrets in LLM apps even in retry logs. Use the JSON formatter tool to inspect sanitized error payloads during development.

Operational checklist before launch

• Define max retries and log every attempt with latency and token count.
• Move batch summarization and embedding jobs off the request thread.
• Set billing alerts at 80% of monthly quota.
• Document which features are safe to degrade when limits hit.
• Test a synthetic 429 in staging with a mocked HTTP client.

These steps align with broader LLMOps ship and operate practice. They also reduce support tickets on apps like Mijar Law Associates where document AI must not block core portal workflows. For greenfield work, API development and custom software development engagements should spec retry behavior in the architecture doc, not as a post-launch patch.

Cost control and retry policy overlap heavily. Aggressive retries during a marketing campaign can burn a monthly LLM budget in hours. Read LLM cost optimization for production apps alongside this guide. The same third-party patterns from third-party API integration retry and backoff apply here. LLM vendors are just expensive, nondeterministic HTTP APIs.

Function-calling flows add another wrinkle. If the model returns tool calls, your retry must replay the whole turn consistently. Partial retries mid-tool-chain produce garbage state. See function calling and tool use with LLMs for design patterns that keep retries safe.

Key Takeaways

• Parse 429 and 503 separately from quota and validation errors before any retry runs.
• Use exponential backoff with jitter and honor Retry-After when the provider sends it.
• Throttle outbound LLM calls with queues and Redis TPM tracking instead of hoping retries save you.
• Attach idempotency keys to user actions so retried completions never duplicate billable work.
• Keep LLM calls off the synchronous web path; return job IDs or stream via SSE for chat UX.
• Log limit events and queue depth early—they predict billing overruns before finance notices.

People Also Ask

How many times should you retry an LLM API call?

Three to five attempts is a sensible default for background jobs with exponential backoff capped at sixty seconds. Interactive chat should retry at most twice or hand off to a queue. Always stop when the error type is insufficient quota or a 400-level validation failure.

What is the difference between RPM and TPM rate limits?

RPM counts HTTP requests per minute regardless of size. TPM counts tokens processed per minute across input and output. A single long document request can hit TPM while RPM still looks healthy. Size workers to both limits.

Should LLM retries be synchronous or queued?

Queue them. Synchronous retries in a web request tie up PHP-FPM workers and frustrate users. Dispatch a job, return 202 Accepted, and notify the client when complete. Synchronous retry belongs only in CLI scripts or admin one-offs.

Does caching reduce LLM rate limit pressure?

Yes. Cache identical prompt hashes when business rules allow it. Embeddings and FAQ answers are ideal cache candidates. Caching does not replace throttling, but it cuts duplicate spend and keeps you under TPM during traffic spikes.

Ship LLM features that survive real traffic

Handle LLM rate limits and retries as infrastructure, not a try/catch afterthought. Throttle before you call, backoff with jitter when the provider pushes back, and fail fast on errors that will never succeed. That combination keeps AI features stable on Laravel 12 and Laravel 13 apps without surprise API bills. If you want help wiring queues, provider failover, or LLMOps monitoring into a production product, contact us or explore AI integration and automation services. You can also browse the portfolio for examples of operational web systems that prioritize reliability over demo-day magic.