SambaStack Prompt Caching - SambaNova Documentation

Prompt caching saves processing time by caching computed results for repeated prompt prefixes – not the prompt text itself, but the work done to process it. When multiple requests share the same opening text – like a system prompt or document – SambaNova serves the cached result instead of reprocessing those tokens from scratch. The first few requests populate the cache on the serving node(s); savings build up as your prefix is observed multiple times. For sustained traffic, hit rates typically reach 90%+ once the prefix has been seen a few times. Prompt caching is a stable (generally available) feature for MiniMax-M2.7 on SambaStack as of the v2.0.2 release. In v2.2.3, prompt caching is also available for MiniMax-M3 as a Preview feature. In v2.2.5, the MiniMax-M3 prompt caching bundles also accept image input. In v2.3, they also accept video input. Requests that contain an image or video, including in an earlier turn, are served without prompt caching. In v2.2.5, prompt caching is also available for GLM-5.3 as a Preview feature, on the glm-5-3-acb-prefix-caching bundle. This bundle accepts text input only. In v2.2.5, prompt caching is also available for gemma-4-31B-it, on the gemma-4-31b-acb-prefix-caching bundle. In v2.3, this bundle also accepts image and video input, and the gemma-4-31b-acb-prefix-caching-32k bundle adds prompt caching at up to 32K context. Requests that contain an image or video, including in an earlier turn, are served without prompt caching. In v2.3, prompt caching is also available for MiniMax-M3.1-Flash as a Preview feature, on the minimax-m3-1-flash-acb-prefix-caching and minimax-m3-1-flash-32k-acb-prefix-caching bundles. These bundles accept text, image, and video input. Requests that contain an image or video, including in an earlier turn, are served without prompt caching.

This page covers how to use prompt caching and interpret usage fields. For model availability, see Supported models and bundles.

When to use prompt caching

Prompt caching delivers the most savings when a large portion of your input is repeated across requests:

Good fit	Poor fit
Long system prompt shared across all user turns	Fully unique inputs per request (no shared prefix)
Repeated few-shot examples or tool definitions	Very short prompts where caching overhead outweighs savings
RAG context re-sent with each query	Single-use documents where the prefix never repeats
Multi-turn conversations with a fixed preamble

If cached_tokens stays at 0 across requests, your inputs do not share a cacheable prefix – review whether your system prompt or context block is stable across calls.

How prompt caching works

Automatic Prefix Caching (APC) is available for MiniMax-M2.7 but must be enabled by a SambaStack administrator as follows:

Deploy a prompt-caching bundle – dyt-minimax-m2p7-32-64-160-192k-pc (32K–192K) or dyt-minimax-m2p7-32k-pc (up to 32K) – or deploy MiniMax-M2.7 directly in a ModelDeployment resource with the minimax-m2p5-dyt-pc-cd model profile.

Set the following environment variables in the ModelDeployment resource:

engineConfig:
  env_vars:
    ENABLE_KV_CACHE_MANAGER: "true"
    KV_CACHE_INCLUDE_STATS_IN_RESPONSE: "true"

The minimax-m2p5-dyt-pc-cd profile name refers to the PEF that MiniMax-M2.5 and MiniMax-M2.7 share, so the m2p5 segment is expected when serving MiniMax-M2.7.

Once enabled, APC detects shared prefixes across requests and reuses cached computations transparently – no changes to your API requests are required. Prompt caching follows the OpenAI standard. Cache hits are detected automatically and reported in the response usage field – no cache_control markers or other request body changes are needed. APC is local to each serving instance. Cached KV state is stored in local DDR and is not transferred across instances. On multi-instance deployments, your prefix is cached independently on each instance after it is served there. The sequence state pool size for MiniMax-M2.7 is 192K.

Settings derived from the PEF

In v2.3, the MiniMax-M3, MiniMax-M3.1-Flash, GLM-5.3, and gemma-4-31B-it prompt caching bundles take their cache settings from the compiled model (PEF):

The engine turns on ENABLE_KV_CACHE_MANAGER and KV_CACHE_INCLUDE_STATS_IN_RESPONSE when the PEF supports prompt caching.
The engine reads the cache page size and the DDR page pool from the PEF.
The router sets smart_routing.block_hash_size to the PEF’s page size.

You can still set these values as overrides, but they must match the PEF:

KV_CACHE_BLOCK_SIZE must equal the PEF’s page size. Any other value stops the engine at startup.
KV_CACHE_NUM_DDR_BLOCKS can reduce the DDR page pool but cannot exceed it. On GLM-5.3, it must equal the PEF’s pool.
A smart_routing.block_hash_size set on the model spec replaces the derived value. The router hashes prompts per cache page, so it must equal the page size.

Enable prompt caching for MiniMax-M3

Deploy minimax-m3-acb-prefix-caching (up to 1M context), or a bundle capped at a shorter context – minimax-m3-32k-acb-prefix-caching, minimax-m3-256k-acb-prefix-caching, minimax-m3-512k-acb-prefix-caching, or minimax-m3-1m-acb-prefix-caching – with continuous batching set to aggregate. See Deploy a configuration.

Set KV_CACHE_NUM_DDR_BLOCKS in the ModelDeployment resource:

engineConfig:
  env_vars:
    KV_CACHE_NUM_DDR_BLOCKS: "1536"

MiniMax-M3 caches prompts in 4,096-token pages. Only whole pages that match an earlier request are reused.

Enable prompt caching for MiniMax-M3.1-Flash

Deploy minimax-m3-1-flash-acb-prefix-caching (up to 1M context) or minimax-m3-1-flash-32k-acb-prefix-caching (up to 32K context) with continuous batching set to aggregate. See Deploy a configuration. The bundles need no prompt caching environment variables or smart_routing.block_hash_size setting. MiniMax-M3.1-Flash caches prompts in 4,096-token pages. Only whole pages that match an earlier request are reused.

Enable prompt caching for GLM-5.3

Deploy the glm-5-3-acb-prefix-caching bundle with continuous batching set to aggregate. See Deploy a configuration. The bundle needs no prompt caching environment variables or smart_routing.block_hash_size setting. In v2.3, GLM-5.3 caches prompts in 4,096-token pages, down from 32,768. Only whole pages that match an earlier request are reused. If your deployment carries the v2.2.5 settings KV_CACHE_BLOCK_SIZE: "32768", KV_CACHE_NUM_DDR_BLOCKS: "64", and block_hash_size: 32768, remove them. A block size of 32,768 does not match the v2.3 PEF and stops the engine at startup.

Enable prompt caching for gemma-4-31B-it

Deploy one of the following bundles with continuous batching set to aggregate. See Deploy a configuration.

gemma-4-31b-acb-prefix-caching: up to 256K context.
gemma-4-31b-acb-prefix-caching-32k: up to 32K context, with up to 64 concurrent requests.

The bundles need no prompt caching environment variables or smart_routing.block_hash_size setting. gemma-4-31B-it caches prompts in 2,048-token pages. Only whole pages that match an earlier request are reused.

Checking cache usage in the response

Every /v1/chat/completions response from a cache-enabled model includes cache token counts in the usage field. The following example shows a second request that shares the same long system prompt as a prior request – the prefix is served from cache rather than reprocessed. Request (second call – cache warm)

# A long, repeated system prompt (1,000+ tokens) triggers APC on the second call.
curl https://<sambastack-host>/v1/chat/completions \
  -H "Authorization: Bearer $SAMBANOVA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "MiniMax-M2.7",
    "messages": [
      {
        "role": "system",
        "content": "You are a technical assistant for SambaNova hardware. ..."
      },
      {
        "role": "user",
        "content": "What is the default batch size for MiniMax-M2.7?"
      }
    ],
    "max_tokens": 100
  }'

Response

{
  "id": "cmpl-3ca55e7f-8fd5-470d-bac6-ee38a7a5307b",
  "object": "chat.completion",
  "created": 1779835973,
  "model": "MiniMax-M2.7",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "The default batch size for MiniMax-M2.7 is 2."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 5797,
    "completion_tokens": 100,
    "total_tokens": 5897,
    "prompt_tokens_details": {
      "cache_creation_tokens": 0,
      "cached_tokens": 4096
    }
  }
}

Field	Description
`cached_tokens`	Input tokens served from the cache. These are billed at the cached input rate.
`cache_creation_tokens`	Input tokens used to populate the cache on this request. Informational only – no additional charge.

Input remaining = prompt_tokens − cached_tokens. This portion is billed at the standard input rate. A cached_tokens value of 0 means no cache hit occurred on that request – it is not an error. This happens on the first request for a given prefix (cache is being populated) or when the input does not share a prefix with any cached entry. For models that do not support prompt caching, cached_tokens is always 0 and billing is unchanged.

Code example

from sambanova import SambaNova

client = SambaNova(
    base_url="https://<sambastack-host>",
    api_key="your-sambanova-api-key",
)

system_prompt = """You are a technical assistant for SambaNova hardware. You have access to
the following reference documentation:

[Insert your long, stable system prompt or document here – the longer and more stable this
prefix, the more tokens are eligible for caching. A prefix must fill at least one cache page
(2,048 or 4,096 tokens, depending on the model) to qualify.]"""

def ask(question):
    response = client.chat.completions.create(
        model="MiniMax-M2.7",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": question},
        ],
    )
    usage = response.usage
    cached = usage.prompt_tokens_details.cached_tokens
    total_prompt = usage.prompt_tokens
    print(f"Cached tokens: {cached} / {total_prompt} prompt tokens")
    print(response.choices[0].message.content)

# First request – populates the cache
ask("What is the default batch size for MiniMax-M2.7?")

# Second request – same prefix, served from cache
ask("What is the maximum sequence length for MiniMax-M2.7?")

from openai import OpenAI

client = OpenAI(
    base_url="https://<sambastack-host>/v1",
    api_key="your-sambanova-api-key",
)

system_prompt = """You are a technical assistant for SambaNova hardware. You have access to
the following reference documentation:

[Insert your long, stable system prompt or document here – the longer and more stable this
prefix, the more tokens are eligible for caching. A prefix must fill at least one cache page
(2,048 or 4,096 tokens, depending on the model) to qualify.]"""

def ask(question):
    response = client.chat.completions.create(
        model="MiniMax-M2.7",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": question},
        ],
    )
    usage = response.usage
    cached = usage.prompt_tokens_details.cached_tokens
    total_prompt = usage.prompt_tokens
    print(f"Cached tokens: {cached} / {total_prompt} prompt tokens")
    print(response.choices[0].message.content)

# First request – populates the cache
ask("What is the default batch size for MiniMax-M2.7?")

# Second request – same prefix, served from cache
ask("What is the maximum sequence length for MiniMax-M2.7?")

Billing impact

Cached tokens are billed at a lower rate than standard input tokens. The billing calculation for a cache-enabled request is:

Token type	Billing
Input remaining (`prompt_tokens` − `cached_tokens`)	Standard input rate
`cached_tokens`	Cached input rate (lower)
`completion_tokens`	Standard output rate (unchanged)

For example, a request with 5,797 total input tokens where 4,096 are served from cache: you pay the standard input rate on 1,701 tokens and the cached input rate on 4,096 tokens. Output billing is unchanged. cache_creation_tokens appears in the response but is informational only – it represents tokens used to populate the cache on this request and carries no additional charge at this time. For models without prompt caching, cached_tokens is always 0, so the formula reduces to the standard calculation with no billing change.

Contact your SambaNova representative for current pricing, including the cached input rate for MiniMax-M2.7.

Limitations

Prompt caching is available on MiniMax-M2.7 and MiniMax-M3, and in Preview on MiniMax-M3.1-Flash and GLM-5.3. On gemma-4-31B-it, it is available on gemma-4-31b-acb-prefix-caching and gemma-4-31b-acb-prefix-caching-32k.
The GLM-5.3 prompt caching bundle accepts text input only. The MiniMax-M3, MiniMax-M3.1-Flash, and gemma-4-31B-it prompt caching bundles also accept image and video input, but requests that contain an image or video, including in an earlier turn, are not cached. Send video as a base64 data: URL.
Custom bundles with prompt caching enabled for Minimax-M2.7 cannot contain other models.
Cache state is local to each serving instance. On multi-instance deployments, your prefix is cached independently on each instance after it is served there.
APC operates on shared prefixes. Requests that diverge before the cached boundary do not get a cache hit.
On MiniMax-M2.7, the maximum cached prefix (sequence state) is 192K tokens.
Cache eviction uses an LRU (least recently used) policy. Cache persistence duration varies with system load and is not guaranteed.
A prefix must fill at least one cache page to qualify for caching: 4,096 tokens on MiniMax-M2.7, MiniMax-M3, MiniMax-M3.1-Flash, and GLM-5.3, and 2,048 tokens on gemma-4-31B-it.
On MiniMax-M3, GLM-5.3, and gemma-4-31B-it, a repetition_penalty other than 1.0 is not supported.

Getting Started

Service Administration

Hardware Administration

Reference Architecture

Resources

Using prompt caching

When to use prompt caching

How prompt caching works

Settings derived from the PEF

Enable prompt caching for MiniMax-M3

Enable prompt caching for MiniMax-M3.1-Flash

Enable prompt caching for GLM-5.3

Enable prompt caching for gemma-4-31B-it

Checking cache usage in the response

Code example

Billing impact

Limitations

​When to use prompt caching

​How prompt caching works

​Settings derived from the PEF

​Enable prompt caching for MiniMax-M3

​Enable prompt caching for MiniMax-M3.1-Flash

​Enable prompt caching for GLM-5.3

​Enable prompt caching for gemma-4-31B-it

​Checking cache usage in the response

​Code example

​Billing impact

​Limitations

When to use prompt caching

How prompt caching works

Settings derived from the PEF

Enable prompt caching for MiniMax-M3

Enable prompt caching for MiniMax-M3.1-Flash

Enable prompt caching for GLM-5.3

Enable prompt caching for gemma-4-31B-it

Checking cache usage in the response

Code example

Billing impact

Limitations