> ## Documentation Index
> Fetch the complete documentation index at: https://sambanova-systems.mintlify.site/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Reduce latency and cost with prompt caching
Prompt caching saves processing time by caching computed results for repeated prompt prefixes – not the prompt text itself, but the work done to process it. When multiple requests share the same opening text – like a system prompt or document – SambaNova serves the cached result instead of reprocessing those tokens from scratch. The first few requests populate the cache on the serving node(s); savings build up as your prefix is observed multiple times. For sustained traffic, hit rates typically reach 90%+ once the prefix has been seen a few times.
Prompt caching is available on **MiniMax-M2.7** on SambaCloud. Automatic Prefix Caching (APC) is enabled by default – no changes to your request are required.
## When to use prompt caching
Prompt caching delivers the most savings when a large portion of your input is repeated across requests:
| Good fit | Poor fit |
| :---------------------------------------------- | :-------------------------------------------------- |
| Long system prompt shared across all user turns | Fully unique inputs per request (no shared prefix) |
| Repeated few-shot examples or tool definitions | Very short prompts where savings are negligible |
| RAG context re-sent with each query | Single-use documents where the prefix never repeats |
| Multi-turn conversations with a fixed preamble | |
If `cached_tokens` stays at `0` across requests, your inputs do not share a cacheable prefix – check whether your system prompt or context block is stable across calls.
## How it works
Automatic Prefix Caching (APC) is always on for MiniMax-M2.7. It detects shared prefixes across requests and reuses cached computations transparently – no API changes required. Keep the leading portion of your messages stable and consistent across requests to maximize cache hits.
A prefix is matched when the leading token sequence of the messages array is identical to a cached request. Any change to the prefix – rewording the system prompt, inserting a message before it, or reordering content – starts a new cache entry. Changing only the user message while keeping the system prompt identical reuses the cached prefix.
Cache state is local to each serving instance. On multi-instance deployments, your prefix is cached independently on each instance after it is served there.
## Read cache usage from the response
Every response includes a `prompt_tokens_details` object showing how many tokens were served from cache:
```json theme={null}
{
"usage": {
"prompt_tokens": 5797,
"completion_tokens": 100,
"total_tokens": 5897,
"prompt_tokens_details": {
"cache_creation_tokens": 0,
"cached_tokens": 4096
}
}
}
```
| Field | Description |
| :---------------------- | :---------------------------------------------------------------------------------- |
| `cached_tokens` | Tokens served from cache. Billed at the cached input rate (lower than standard). |
| `cache_creation_tokens` | Tokens written to cache on this request. Informational only – no additional charge. |
`cache_creation_tokens` is non-zero on the first request that builds the cache entry and `0` on subsequent cache hits. The number of non-cached input tokens is `prompt_tokens - cached_tokens`.
## Billing
Cached tokens are billed at a lower rate than standard input tokens. The full pricing for your model – including the cached input rate and cache write rate – is available via the `/v1/models` endpoint and on the [SambaCloud pricing page](https://cloud.sambanova.ai/pricing).
```bash theme={null}
curl https://cloud.sambanova.ai/v1/models \
-H "Authorization: Bearer $SAMBANOVA_API_KEY" \
| jq '.data[] | select(.id == "MiniMax-M2.7") | .pricing'
```
The billing formula for a cache-enabled request:
```
cost = (prompt_tokens − cached_tokens) × standard_input_rate
+ cached_tokens × cached_input_rate
+ completion_tokens × output_rate
```
For example, a request with 5797 total input tokens where 4096 are served from cache: only 1701 tokens are billed at the standard input rate, and 4096 at the lower cached rate. Output billing is unchanged.
## Code example
Caching activates only when the shared prefix reaches 4096 tokens. The system prompt below is shorter for readability – replace it with your actual document to observe cache hits in practice.
```python Python (SambaNova) theme={null}
from sambanova import SambaNova
client = SambaNova(
base_url="your-sambanova-base-url",
api_key="your-sambanova-api-key",
)
system_prompt = """You are a financial analyst assistant. You have access to the following
quarterly earnings report:
Fiscal Q3 2024 Earnings Report – Your Company
Revenue: Total revenue for Q3 2024 was $4.2 billion, up 12% year-over-year. Product revenue
was $3.1 billion (+9% YoY) and services revenue was $1.1 billion (+21% YoY).
Gross Margin: GAAP gross margin was 68.4%, up from 65.2% in Q3 2023. Non-GAAP gross margin
was 71.1%.
Operating Income: GAAP operating income was $820 million (19.5% margin). Non-GAAP operating
income was $1.05 billion (25.0% margin).
Net Income: GAAP net income was $710 million, or $1.42 per diluted share. Non-GAAP net income
was $910 million, or $1.82 per diluted share.
Cash: Cash, cash equivalents, and short-term investments totaled $12.3 billion at quarter end.
Free cash flow was $980 million.
Outlook: Q4 2024 revenue guidance is $4.4–$4.6 billion.
Replace this content with your actual document. The longer and more stable your system prompt,
the more tokens are eligible for caching."""
def ask(question):
response = client.chat.completions.create(
model="MiniMax-M2.7",
messages=[
{"role": "system", "content": system_prompt},
{"role": "user", "content": question},
],
)
usage = response.usage
cached = usage.prompt_tokens_details.cached_tokens
total_prompt = usage.prompt_tokens
print(f"Cached tokens: {cached} / {total_prompt} prompt tokens")
print(response.choices[0].message.content)
# First request – populates the cache
ask("What was the revenue in Q3?")
# Second request – same prefix, served from cache
ask("What was the gross margin in Q3?")
```
```python Python (OpenAI) theme={null}
from openai import OpenAI
client = OpenAI(
base_url="your-sambanova-base-url",
api_key="your-sambanova-api-key",
)
system_prompt = """You are a financial analyst assistant. You have access to the following
quarterly earnings report:
Fiscal Q3 2024 Earnings Report – Your Company
Revenue: Total revenue for Q3 2024 was $4.2 billion, up 12% year-over-year. Product revenue
was $3.1 billion (+9% YoY) and services revenue was $1.1 billion (+21% YoY).
Gross Margin: GAAP gross margin was 68.4%, up from 65.2% in Q3 2023. Non-GAAP gross margin
was 71.1%.
Operating Income: GAAP operating income was $820 million (19.5% margin). Non-GAAP operating
income was $1.05 billion (25.0% margin).
Net Income: GAAP net income was $710 million, or $1.42 per diluted share. Non-GAAP net income
was $910 million, or $1.82 per diluted share.
Cash: Cash, cash equivalents, and short-term investments totaled $12.3 billion at quarter end.
Free cash flow was $980 million.
Outlook: Q4 2024 revenue guidance is $4.4–$4.6 billion.
Replace this content with your actual document. The longer and more stable your system prompt,
the more tokens are eligible for caching."""
def ask(question):
response = client.chat.completions.create(
model="MiniMax-M2.7",
messages=[
{"role": "system", "content": system_prompt},
{"role": "user", "content": question},
],
)
usage = response.usage
cached = usage.prompt_tokens_details.cached_tokens
total_prompt = usage.prompt_tokens
print(f"Cached tokens: {cached} / {total_prompt} prompt tokens")
print(response.choices[0].message.content)
# First request – populates the cache
ask("What was the revenue in Q3?")
# Second request – same prefix, served from cache
ask("What was the gross margin in Q3?")
```
## Limitations
* Prompt caching is available on **MiniMax-M2.7** only. Other models return `cached_tokens: 0`.
* Cache state is local to a single node and is not shared across nodes.
* Maximum cacheable prefix length: **192000 tokens**.
* A prefix must contain at least **4096 tokens** to qualify for caching.
* Cache eviction uses an **LRU (least recently used)** policy. Cache persistence varies with system load and is not guaranteed.