Prompt Caching to Cut LLM Costs

By Kokil Thapa | Last reviewed: September 2026

Production LLM bills spike when every request resends the same system prompt, tool schemas, and RAG documents. Prompt caching to cut LLM costs solves that by storing a stable prefix on the provider side and charging a lower rate for cache hits. On client projects with AI integration and automation, I treat cache design as architecture work—not a billing afterthought. This guide covers how caching works, what each major provider charges, and how to structure prompts so savings actually show up in your invoice.

What Is Prompt Caching and How Does It Cut LLM Costs?

Prompt caching is a provider feature that stores a computed representation of an initial token sequence. Later requests with the same prefix reuse that work. You still pay for tokens, but cached input tokens cost far less than fresh ones.

Think of it like HTTP caching for text. The first request pays full price to populate the cache. Follow-up requests within the TTL window pay a small read fee instead of recomputing attention over thousands of tokens.

This differs from application-level caching of final answers. Prompt caching speeds up and cheapens the input side. Answer caching—storing completed responses in Redis—can stack on top for identical user queries. See Redis caching patterns for web apps for that layer.

Providers implement this differently. Anthropic exposes explicit cache_control breakpoints. OpenAI applies automatic prefix caching on supported models. Google Cloud offers context caching on Vertex AI. The billing labels vary, but the pattern is the same: identical prefix, lower marginal cost.

How Do You Structure Prompts for Maximum Cache Hits?

Cache hits require byte-identical prefixes. A single changed space, reordered JSON key, or timestamp in the system prompt breaks the match. That sounds strict. In practice, most teams fail here—not at the API call layer.

Put static content first, dynamic content last

Order your messages from least to most variable:

1. System instructions and persona rules
2. Tool definitions and JSON schemas
3. Few-shot examples that rarely change
4. Retrieved RAG chunks for the current session
5. The user's latest message

Only the tail should change per request. Everything above it becomes your cacheable block. This mirrors how prompt engineering best practices already recommend separating instructions from user input.

Use explicit cache breakpoints on Anthropic

Anthropic's Messages API lets you mark cacheable blocks with cache_control:

{
  "model": "claude-sonnet-4-20250514",
  "max_tokens": 1024,
  "system": [
    {
      "type": "text",
      "text": "You are a legal intake assistant for Nepal notary services...",
      "cache_control": { "type": "ephemeral" }
    }
  ],
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "FULL KNOWLEDGE BASE DOCUMENT HERE...",
          "cache_control": { "type": "ephemeral" }
        },
        {
          "type": "text",
          "text": "User question: What documents are needed for attestation?"
        }
      ]
    }
  ]
}

Mark large, stable blocks—system prompt, tool schemas, reference docs—with ephemeral cache control. Keep the final user turn uncached. Official details live in Anthropic's prompt caching documentation.

Stabilise tool and schema payloads

Function-calling apps often rebuild tool JSON on every request. Don't. Serialize tools once at boot or deploy time. Store the exact string in config. Pass the same bytes every call.

I've seen Laravel apps lose cache hits because json_encode emitted keys in different orders across PHP versions. Use JSON_UNESCAPED_SLASHES consistently. Pin key order manually for critical blocks. Validate payloads with a JSON formatter during CI so whitespace drift gets caught early.

Session-level RAG caching

For chat over a fixed document set—client uploads, case files, product manuals—attach retrieved chunks as a cached block for the session duration. The user's follow-up questions then hit cache on the document prefix. Only the new question token count grows.

This pattern works well on legal-tech portals where users ask five to ten questions against the same PDF bundle. The first question pays to cache the context. Questions two through ten are much cheaper.

How Much Can Prompt Caching Actually Save on LLM Bills?

Savings depend on prefix size, request volume, and provider pricing. Small prefixes on low-traffic apps save pennies. Large RAG prefixes on chat apps can cut input costs by half or more.

Here is a simplified example. Suppose your app sends 8,000 tokens of system + tool + document context on every request. Input costs $3 per million tokens. You handle 10,000 requests per month.

• Without caching: 8,000 × 10,000 = 80M input tokens → $240/month on context alone
• With 90% cache hit rate: 8,000 cache writes once, then 9,000 hits at ~$0.30/M cached rate → roughly $30–$50/month on the same context block

That is an order-of-magnitude difference on the prefix. Output tokens still bill at full rate. Caching is not a magic zero bill. It targets the repeated input problem that RAG and agent apps suffer from.

For broader FinOps context—budgets, alerts, tagging—pair this with LLM cost optimization for production apps and AI rate limits and cost optimization.

Check current numbers on each vendor's pricing page before you budget. Rates change quarterly. The architectural lesson stays stable: big static prefix plus high repeat rate equals real savings.

How Do You Implement Prompt Caching in a Production Laravel App?

Most of my production LLM integrations run on Laravel 12 or 13 with PHP 8.3+. Caching logic belongs in a dedicated service class—not scattered across controllers. Keep provider details behind an interface so you can swap vendors without rewriting prompt assembly.

Step 1: Centralise prompt assembly

<?php

namespace App\Services\Llm;

final class PromptBuilder
{
    private string $systemPrompt;
    private string $toolsJson;

    public function __construct()
    {
        $this->systemPrompt = (string) config('llm.system_prompt');
        $this->toolsJson = (string) config('llm.tools_json');
    }

    public function forChat(string $sessionDocs, string $userMessage): array
    {
        return [
            'system' => [
                [
                    'type' => 'text',
                    'text' => $this->systemPrompt,
                    'cache_control' => ['type' => 'ephemeral'],
                ],
            ],
            'messages' => [
                [
                    'role' => 'user',
                    'content' => [
                        [
                            'type' => 'text',
                            'text' => $sessionDocs,
                            'cache_control' => ['type' => 'ephemeral'],
                        ],
                        [
                            'type' => 'text',
                            'text' => $userMessage,
                        ],
                    ],
                ],
            ],
        ];
    }
}

Load system_prompt and tools_json from config files deployed with your release. Never interpolate now(), request IDs, or user names into cached blocks.

Step 2: Log cache metrics from API responses

Anthropic returns cache_creation_input_tokens and cache_read_input_tokens in usage metadata. OpenAI returns prompt_tokens_details.cached_tokens. Log these fields on every call.

Log::info('llm.usage', [
    'model' => $response['model'],
    'cache_read' => $response['usage']['cache_read_input_tokens'] ?? 0,
    'cache_write' => $response['usage']['cache_creation_input_tokens'] ?? 0,
    'input' => $response['usage']['input_tokens'] ?? 0,
    'output' => $response['usage']['output_tokens'] ?? 0,
]);

Build a weekly report: cache hit ratio, cost per conversation, cost per resolved ticket. Without these numbers, you are guessing. This is core LLMOps work, not optional telemetry.

Step 3: Align TTL with user behaviour

Anthropic's default five-minute TTL refreshes on each hit. Design chat UX so follow-up questions arrive inside that window when possible. For longer gaps, accept a cache miss and move on.

Vertex AI lets you set TTL in hours. That suits batch workflows—nightly report generation, document review queues—more than real-time chat. Pick the provider feature that matches your traffic pattern.

Step 4: Queue workers and idempotent prefixes

Background jobs that call LLMs benefit enormously from caching. A job processing 500 similar support tickets can share one cached policy manual. Ensure each job passes the same prefix string. Normalise line endings. Strip BOM characters. Hash the prefix in logs to verify consistency across workers.

For API design patterns around efficient backends, see API development and function calling and tool use with LLMs.

What Mistakes Break Prompt Caching and Waste Money?

Teams enable caching, see one good invoice, then wonder why savings disappear. These failure modes show up repeatedly.

Dynamic content inside cached blocks

Putting {{ current_date }} or the user's name in the system prompt kills cache hits on every request. Move all personalisation to the uncached tail. If the model needs the user's name, append it just before the question—not in the cached persona block.

Unstable serialisation

Rebuilding JSON tool arrays from database rows without fixed ordering changes the prefix hash. Pin serialisation. Add a unit test that asserts the tool JSON string equals a committed fixture file.

Caching too little

Prefixes under the provider minimum—often 1,024 tokens—do not cache. Pad is not an option; providers reject gaming. Instead, merge small fragments. Combine system prompt and tools into one cached block if each alone is too small.

Ignoring cache write costs

The first request pays a cache write premium on top of normal input tokens. Low-traffic endpoints with unique prefixes every call can cost more with caching enabled. Cache write fees hurt when hit rate is low. Measure before enabling globally.

Mixing caching with PII-heavy context

Cached blocks live on the provider's infrastructure for the TTL window. Do not cache raw passport numbers, bank details, or privileged legal content unless your data-processing agreement covers it. Redact or tokenise sensitive fields first. Read protecting PII and secrets in LLM apps before caching client documents on a shared legal-tech portal.

Skipping evals after prompt changes

Moving content between cached and uncached blocks can shift model behaviour. Re-run evals when you restructure prompts. Quality regressions erase dollar savings fast. Use the workflow in how to evaluate LLM outputs.

When Should You Choose Prompt Caching Over Fine-Tuning or Smaller Models?

Caching is one lever in a cost stack. It is not always the best lever.

Choose prompt caching when: you have large repeated context, multi-turn chat, or agent loops that resend the same tools and policies. RAG apps with session-scoped documents are the sweet spot. Integration effort is low—often a few hours of prompt restructuring.

Choose a smaller or self-hosted model when: tasks are simple classification or extraction and quality holds up on a cheaper tier. See fine-tuning vs prompt engineering and self-hosting an LLM for that path.

Choose fine-tuning when: you need consistent output format or domain tone that prompting cannot stabilise—but the training data pipeline is ready and you accept maintenance cost. Caching and fine-tuning can combine. A fine-tuned model still benefits from cached system prompts.

On a client portal with document Q&A, caching session documents cut repeated input cost without touching model choice. That is typical for Nepal legal-tech workloads: moderate traffic, heavy context, high repeat questions within a session.

Also consider long-context LLM strategies before stuffing ever more tokens into cache. Sometimes chunking and retrieval beat a 100K-token cached block on both cost and accuracy.

Key Takeaways

• Put static system prompts, tools, and session documents in cacheable prefix blocks; keep user messages in the dynamic tail.
• Require byte-identical prefixes—pin JSON serialisation, ban timestamps in cached blocks, and hash prefixes in CI.
• Log cache_read_input_tokens or cached_tokens on every API call and track hit ratio weekly.
• Expect the biggest savings on RAG chat and agent apps that resend thousands of tokens per turn.
• Watch cache write fees on low-traffic endpoints with unique prefixes—they can cost more than no caching.
• Never cache PII-heavy content without reviewing your provider's data retention terms and redaction policy.

People Also Ask

Does prompt caching reduce output token costs?

No. Prompt caching discounts input tokens that match a stored prefix. Output tokens always bill at the standard generation rate. Savings come from not reprocessing large input context on every request—not from cheaper completions.

How long does an LLM prompt cache last?

It depends on the provider. Anthropic's ephemeral cache typically expires after five minutes of inactivity but refreshes on each cache hit. Google Vertex AI context caches support configurable TTLs measured in hours. OpenAI manages TTL internally—check usage metadata rather than assuming a fixed window.

Is prompt caching the same as storing responses in Redis?

No. Prompt caching operates on the provider side at the model input layer. Redis response caching stores final answers for identical queries. They complement each other. Use prompt caching for repeated context; use Redis when the same question gets asked many times across users.

Which apps benefit most from prompt caching?

Multi-turn chatbots, RAG assistants, and agent workflows that resend tool definitions and reference docs on every call benefit most. Simple single-turn classifiers with short prompts and low volume rarely justify the engineering effort unless traffic is very high.

Ship Caching Before Your LLM Bill Becomes a Problem

Prompt caching to cut LLM costs is the fastest win most production teams skip. Restructure prompts once, log cache metrics, and validate hit rates after each deploy. Pair caching with prompt engineering discipline, LLMOps monitoring, and sensible model selection. Your prefix is already being sent—make the provider charge you less for it.

Need help wiring caching into a Laravel app, legal-tech chatbot, or customer-support agent? Contact us to review your prompt architecture and estimate real savings before your next billing cycle.