Prompt Caching Deep Dive: Cutting Latency & Cost

Your RAG endpoint sends the same 8,000-token system prompt — instructions, tool schemas, a fat style guide — on every single request, then appends a 20-token user question. You are re-paying full input price to re-read an identical preamble thousands of times a day. The model doesn't need to re-read what hasn't changed. Prompt caching makes it stop, and the savings on a high-volume path are not marginal — they're structural.

Core Architectural Concepts & Trade-offs

Prompt caching stores the internal representation of a prompt prefix so subsequent requests that share that exact prefix skip recomputing it. You mark cacheable boundaries with cache_control breakpoints; on a cache hit, those input tokens bill at roughly a tenth of the normal rate and skip the compute, which cuts cost and time-to-first-token. On a high-traffic endpoint with a big static preamble, that's the single highest-ROI change you can make — and it requires zero quality trade-off, unlike compaction (Lesson 1) which is lossy.

The mechanic that governs everything is prefix matching: the cache keys on an exact-match prefix from the very start of the prompt. Everything up to a breakpoint must be byte-identical to hit. The engineering consequence is an ordering discipline — static content first, dynamic content last. System prompt, then tool definitions, then stable reference documents, then the volatile conversation and the user's turn. Put a timestamp or a per-request ID near the top and you've invalidated the entire cache below it; the prefix no longer matches and you pay full price for all of it.

Caches are ephemeral, and that shapes the workload they help. The cached prefix has a short time-to-live — on the order of a few minutes — refreshed on each hit. So caching is a windfall for bursty, repeated traffic (multi-turn sessions, a flood of requests over the same documents) and does nothing for cold, one-off calls spaced hours apart. Know your traffic shape before you count the savings: a chat session reuses the prefix every turn; a nightly batch job hits a cold cache every time.

Two interactions to design around. Caching pairs naturally with cheap-tier routing — a Haiku 4.5 extraction loop (Lesson 3) over a cached document prefix is about as cheap as production inference gets. And it constrains compaction: summarizing or editing early conversation rewrites the prefix and breaks the cache below it, so compact at boundaries where you were going to invalidate anyway, never mid-burst inside a tight cached loop. The cheapest token is the one you don't recompute — but only if you keep the prefix stable.

Caching the Static Prefix

Mark the stable preamble — tools and system — with a breakpoint. The volatile user turn stays uncached at the end.

// Static first, dynamic last. The breakpoint caches everything above it.
import Anthropic from "@anthropic-ai/sdk"

const anthropic = new Anthropic()

// Stable across every request → cache it once, reuse it for ~minutes.
const TOOLS = [/* …large, stable tool schemas… */]
const SYSTEM_PREAMBLE = LONG_INSTRUCTIONS + STYLE_GUIDE   // ~8K tokens

export async function answer(question: string) {
  return anthropic.messages.create({
    model: "claude-haiku-4-5",
    max_tokens: 1024,
    tools: TOOLS,
    system: [
      {
        type: "text",
        text: SYSTEM_PREAMBLE,
        // Breakpoint: cache the system + tools prefix above this point.
        cache_control: { type: "ephemeral" },
      },
    ],
    // Dynamic tail — never cached, byte-changes every call. Always LAST.
    messages: [{ role: "user", content: question }],
  })
}

// First call: cache write (slightly higher). Every call for the next few
// minutes: cache hit → ~90% off the prefix tokens + faster first token.

Check the response's usage fields — cache_creation_input_tokens on the write, cache_read_input_tokens on the hits — to confirm you're actually landing in the cache and not silently paying full freight.

The Economics of a Stable Prefix

Caching makes single-agent calls cheap; the next module scales out to many agents at once — but first, safety: constitutional AI & guardrails. Or browse the full roadmap.