Why a bigger semantic-layer prompt can cost less than repeated MCP retrieval

A common data-agent design rule sounds obvious: do not send the model a large semantic layer; expose a search tool and fetch only the tables needed for each question. Prompt caching makes that rule incomplete.

Both Anthropic and OpenAI cache prefixes—the unchanged content at the beginning of a request. Anthropic says its cached prefix can include tools, the system prompt and messages up to the chosen breakpoint. OpenAI says its rendered context includes instructions, tool definitions and conversation history, but reuse requires a matching prefix. In other words, a large, stable block of metric definitions can become cheap on repeated calls, while a retrieval loop can keep adding uncached tool calls and responses.

The layout matters more than the raw size

For a text-to-SQL agent, the cache-friendly order is straightforward:

1. Put stable policy, schema and certified metric definitions first.
2. Keep tool definitions stable in name, schema and order.
3. Put volatile user questions and recent conversation after the reusable prefix.
4. Measure cache reads in production rather than assuming the provider found a match.

OpenAI warns that changing tools, model settings or any earlier prefix content can prevent reuse from the first changed token onward. It also documents a minimum cacheable prompt length of 1,024 tokens for GPT-5.6 and later. Anthropic offers automatic caching or explicit breakpoints; its default cache lifetime is five minutes, with a one-hour option at additional cost.

That means a semantic layer is not automatically cacheable just because it is large. Reordering tools on every request, injecting timestamps near the front, or rebuilding definitions nondeterministically can turn the same logical context into a different prefix. Teams should serialize stable context deterministically and watch actual cache-read usage.

What retrieval is still for

Caching does not remove the need for retrieval. A full enterprise catalog may exceed the useful context window; access controls may require per-user filtering; and definitions can change between deployments. Retrieval remains the right tool for narrowing a very large or permission-sensitive corpus.

The useful distinction is therefore not “prompt versus tools.” It is stable context versus variable context. Cache the shared rules and compact definitions that many requests reuse. Retrieve the small, user-specific or fast-changing remainder. Then keep the SQL execution role read-only and enforce authorization outside the model; neither provider describes prompt caching as an access-control mechanism.

The practical test is an A/B run with the same question set: compare a retrieved top-k context path against a deterministic cached prefix, tracking answer quality, cache-hit rate, uncached input tokens, tool-call tokens and end-to-end latency. Without those five measurements, “smaller prompt” is only an architectural preference—not a cost result.