Prompt caching saves processing time by caching computed results for repeated prompt prefixes – not the prompt text itself, but the work done to process it. When multiple requests share the same opening text – like a system prompt or document – SambaNova serves the cached result instead of reprocessing those tokens from scratch. The first few requests populate the cache on the serving node(s); savings build up as your prefix is observed multiple times. For sustained traffic, hit rates typically reach 90%+ once the prefix has been seen a few times.
Prompt caching is available on MiniMax-M2.7 on SambaCloud. Automatic Prefix Caching (APC) is enabled by default – no changes to your request are required.
Prompt caching delivers the most savings when a large portion of your input is repeated across requests:
Good fit
Poor fit
Long system prompt shared across all user turns
Fully unique inputs per request (no shared prefix)
Repeated few-shot examples or tool definitions
Very short prompts where savings are negligible
RAG context re-sent with each query
Single-use documents where the prefix never repeats
Multi-turn conversations with a fixed preamble
If cached_tokens stays at 0 across requests, your inputs do not share a cacheable prefix – check whether your system prompt or context block is stable across calls.
Automatic Prefix Caching (APC) is always on for MiniMax-M2.7. It detects shared prefixes across requests and reuses cached computations transparently – no API changes required. Keep the leading portion of your messages stable and consistent across requests to maximize cache hits.A prefix is matched when the leading token sequence of the messages array is identical to a cached request. Any change to the prefix – rewording the system prompt, inserting a message before it, or reordering content – starts a new cache entry. Changing only the user message while keeping the system prompt identical reuses the cached prefix.Cache state is local to each serving instance. On multi-instance deployments, your prefix is cached independently on each instance after it is served there.
Tokens served from cache. Billed at the cached input rate (lower than standard).
cache_creation_tokens
Tokens written to cache on this request. Informational only – no additional charge.
cache_creation_tokens is non-zero on the first request that builds the cache entry and 0 on subsequent cache hits. The number of non-cached input tokens is prompt_tokens - cached_tokens.
Cached tokens are billed at a lower rate than standard input tokens. The full pricing for your model – including the cached input rate and cache write rate – is available via the /v1/models endpoint and on the SambaCloud pricing page.
For example, a request with 5797 total input tokens where 4096 are served from cache: only 1701 tokens are billed at the standard input rate, and 4096 at the lower cached rate. Output billing is unchanged.
Caching activates only when the shared prefix reaches 4096 tokens. The system prompt below is shorter for readability – replace it with your actual document to observe cache hits in practice.
from sambanova import SambaNovaclient = SambaNova( base_url="your-sambanova-base-url", api_key="your-sambanova-api-key",)system_prompt = """You are a financial analyst assistant. You have access to the followingquarterly earnings report:Fiscal Q3 2024 Earnings Report – Your CompanyRevenue: Total revenue for Q3 2024 was $4.2 billion, up 12% year-over-year. Product revenuewas $3.1 billion (+9% YoY) and services revenue was $1.1 billion (+21% YoY).Gross Margin: GAAP gross margin was 68.4%, up from 65.2% in Q3 2023. Non-GAAP gross marginwas 71.1%.Operating Income: GAAP operating income was $820 million (19.5% margin). Non-GAAP operatingincome was $1.05 billion (25.0% margin).Net Income: GAAP net income was $710 million, or $1.42 per diluted share. Non-GAAP net incomewas $910 million, or $1.82 per diluted share.Cash: Cash, cash equivalents, and short-term investments totaled $12.3 billion at quarter end.Free cash flow was $980 million.Outlook: Q4 2024 revenue guidance is $4.4–$4.6 billion.Replace this content with your actual document. The longer and more stable your system prompt,the more tokens are eligible for caching."""def ask(question): response = client.chat.completions.create( model="MiniMax-M2.7", messages=[ {"role": "system", "content": system_prompt}, {"role": "user", "content": question}, ], ) usage = response.usage cached = usage.prompt_tokens_details.cached_tokens total_prompt = usage.prompt_tokens print(f"Cached tokens: {cached} / {total_prompt} prompt tokens") print(response.choices[0].message.content)# First request – populates the cacheask("What was the revenue in Q3?")# Second request – same prefix, served from cacheask("What was the gross margin in Q3?")
from openai import OpenAIclient = OpenAI( base_url="your-sambanova-base-url", api_key="your-sambanova-api-key",)system_prompt = """You are a financial analyst assistant. You have access to the followingquarterly earnings report:Fiscal Q3 2024 Earnings Report – Your CompanyRevenue: Total revenue for Q3 2024 was $4.2 billion, up 12% year-over-year. Product revenuewas $3.1 billion (+9% YoY) and services revenue was $1.1 billion (+21% YoY).Gross Margin: GAAP gross margin was 68.4%, up from 65.2% in Q3 2023. Non-GAAP gross marginwas 71.1%.Operating Income: GAAP operating income was $820 million (19.5% margin). Non-GAAP operatingincome was $1.05 billion (25.0% margin).Net Income: GAAP net income was $710 million, or $1.42 per diluted share. Non-GAAP net incomewas $910 million, or $1.82 per diluted share.Cash: Cash, cash equivalents, and short-term investments totaled $12.3 billion at quarter end.Free cash flow was $980 million.Outlook: Q4 2024 revenue guidance is $4.4–$4.6 billion.Replace this content with your actual document. The longer and more stable your system prompt,the more tokens are eligible for caching."""def ask(question): response = client.chat.completions.create( model="MiniMax-M2.7", messages=[ {"role": "system", "content": system_prompt}, {"role": "user", "content": question}, ], ) usage = response.usage cached = usage.prompt_tokens_details.cached_tokens total_prompt = usage.prompt_tokens print(f"Cached tokens: {cached} / {total_prompt} prompt tokens") print(response.choices[0].message.content)# First request – populates the cacheask("What was the revenue in Q3?")# Second request – same prefix, served from cacheask("What was the gross margin in Q3?")
