SambaStack Speculative Decoding Deployment

SambaStack allows you to use custom draft models with both user-provided checkpoints and SambaNova-provided checkpoints to improve inference performance in certain deployment scenarios. These draft models enable speculative decoding, which can reduce latency and increase throughput when configured appropriately. This page explains how to deploy models using speculative decoding. It covers when speculative decoding is appropriate, how to validate compatibility between draft and target models, and how to configure custom draft checkpoints.

If you are unfamiliar with models, model profiles, model bundles, or model deployment, see the Deploying models and bundles page.

What is speculative decoding

Speculative decoding is an inference acceleration technique that pairs a fast, lightweight “draft” model with a larger, higher-quality “target” model:

The draft model proposes multiple next tokens
The target model quickly verifies or rejects them in parallel
This approach significantly speeds up inference decode time, especially when the draft model is closely “aligned” to the target model

Key benefit: Speculative decoding accelerates inference without affecting the output distributions (and thereby accuracy) of the target model.

When NOT to use speculative decoding

Speculative decoding is most effective when the draft model can predict a meaningful portion of the target model’s next tokens. In some situations, however, it can provide little benefit–or even reduce overall performance:

Scenario	Why it hurts performance
Long inputs with short outputs	Both the draft and target models must process the entire input before generation begins. The added overhead of running the draft model can outweigh any speedup.
Very large gaps between model sizes	When the draft model is much smaller (or otherwise poorly aligned) compared to the target model, acceptance rates tend to drop. Low acceptance rates mean more rejected tokens, more corrective forward passes on the target model, and ultimately decreased performance.

Using custom draft checkpoints for speculative decoding

Some model profiles in SambaStack support speculative decoding. When a target model uses speculative decoding, both the draft and target models must be deployed together, and the checkpoints for either model may be customized. This section walks through the full workflow for deploying a custom draft checkpoint, including:

How to identify which profiles and bundles use speculative decoding
How to validate draft–target compatibility
How to update your ModelBundle

Workflow overview

Deploying a custom draft checkpoint for speculative decoding follows these high-level steps:

Step	Action
1	Confirm your deployment uses speculative decoding by checking the model profile and the bundle’s `specDecodingPairs`
2	Convert your custom draft checkpoint into the SambaNova-compatible format
3	Validate compatibility between your draft and target checkpoints (recommended)
4	Upload the converted draft checkpoint to your GCS bucket, or place it on an NFS location that is accessible and readable by your cluster.
5	Reference your draft checkpoint, either by overriding the existing draft model’s checkpoint or by registering a new `Model`
6	Update your `ModelBundle` to reference the custom draft checkpoint

Steps to deploy a custom draft checkpoint

Confirm your deployment uses speculative decoding

Speculative decoding requires a target model whose ModelProfile supports it, together with a draft model deployed alongside it. A profile supports speculative decoding when its spec.features includes speculative_decoding:

kubectl -n <namespace> get modelprofiles -o json

The pairing itself is declared in the ModelBundle through spec.specDecodingPairs. In the example below, meta-llama-3-3-70b-instruct is the target and meta-llama-3-2-1b-instruct is the draft. A third model, gemma-4-31b-it, is bundled alongside them and does not use speculative decoding.

apiVersion: sambanova.ai/v1alpha1
kind: ModelBundle
metadata:
  name: gemma4-31-llama70-sd-1b
spec:
  modelConfigs:
    - model: gemma-4-31b-it:1
      profile: gemma4-31b-dyt
    - model: meta-llama-3-3-70b-instruct:1
      profile: llama-3p1-70b-sd
    - model: meta-llama-3-2-1b-instruct:1
      profile: llama-3p1-1b
      modelSettings:
        routable: false
  specDecodingPairs:
    - draft: meta-llama-3-2-1b-instruct
      target: meta-llama-3-3-70b-instruct

In the above example:

meta-llama-3-3-70b-instruct is the target model. It uses the llama-3p1-70b-sd profile, whose PEFs are compiled for speculative decoding.
meta-llama-3-2-1b-instruct is the draft model. It is marked routable: false, so clients cannot address it directly.
spec.specDecodingPairs declares the pairing between them.
gemma-4-31b-it is unrelated to the pairing and is served normally.

Note the following:

Both the target and the draft model must appear as entries in spec.modelConfigs.
The draft model should be non-routable, since requests are sent to the target model.
Entries in specDecodingPairs reference models by their bare Model resource name, without an architecture or version suffix.

Speculative decoding fields

Field	Required	Description
`target`	Yes	Resource name of the target model, specified without an architecture or version suffix.
`draft`	Yes	Resource name of the draft model, specified without an architecture or version suffix.
`experts`	No	Restricts the pairing to specific experts. Omit this field to apply the pairing to all of the target model’s experts.

Convert your custom draft checkpoint

If you are replacing the draft model’s checkpoint with your own fine-tuned or custom version, convert it into the SambaNova-compatible format using the Checkpoint Conversion Tool.

If you have not already set up the tool, follow the Download and set up instructions on the Checkpoint Conversion Tool documentation page.
Then convert the draft checkpoint using the steps in Convert and validate checkpoint.

SambaStack-provided checkpoints are already in the correct format. If you are using a SambaStack-provided draft checkpoint, skip this step.

Validate draft–target checkpoint compatibility

When deploying custom checkpoints with speculative decoding, verify that a draft checkpoint is compatible with the target checkpoint. This helps prevent unexpected errors during inference.The Checkpoint Conversion Tool provides a built-in utility that checks draft–target compatibility. Use the validation utility to confirm:

The draft and target models align structurally
Their tokenizers are compatible
Speculative decoding can run safely with this pair

Command template

docker run -v $HOST_WORKING_DIR:$DOCKER_WORKING_DIR --rm -it \
    --platform linux/amd64 \
    $IMAGE_NAME \
    validate-sd \
    --target_model $TARGET_MODEL \
    --target_checkpoint_path "$DOCKER_WORKING_DIR/$TARGET_CHECKPOINT_DIR" \
    --draft_model $DRAFT_MODEL \
    --draft_checkpoint_path "$DOCKER_WORKING_DIR/$DRAFT_CHECKPOINT_DIR" \
    --server $SERVER \
    --cache_location $CACHE_LOCATION

Parameters

These are the primary input flags to the validate-sd command:

Flag / Variable	Type	Description
`--target_model` / `TARGET_MODEL`	`str`	The target model family to validate against (e.g., `"llama3-70b"`). This should match the model family used in your target deployment.
`--target_checkpoint_path`	`str`	Path inside the container to the directory containing the converted target checkpoint. Typically `$DOCKER_WORKING_DIR/<subdir>` mounted from `HOST_WORKING_DIR`.
`--draft_model` / `DRAFT_MODEL`	`str`	The draft model family to validate (e.g., `"llama3-8b"`). This should match the model family used as the draft model in your speculative decoding configuration.
`--draft_checkpoint_path`	`str`	Path inside the container to the directory containing the converted draft checkpoint. Typically `$DOCKER_WORKING_DIR/<subdir>` mounted from `HOST_WORKING_DIR`.
`--server` / `SERVER`	`str`	Source of serving metadata. Can be `embedded`, a local path, or a remote URL such as `https://api.sambanova.ai/`. Typically this is the base endpoint URL for your SambaStack instance.
`--cache_location` / `CACHE_LOCATION`	`str`	Location (inside the container) where serving metadata/configs are stored. Usually a subdirectory of `$DOCKER_WORKING_DIR` and visible on the host under `$HOST_WORKING_DIR/$CACHE_LOCATION`.

Host-level variables

In addition to the flags above, you will typically set the following environment variables for the Docker command:

Variable	Description
`HOST_WORKING_DIR`	Directory on the host that contains your converted target and draft checkpoints. Must be writable.
`DOCKER_WORKING_DIR`	Directory inside the container where `HOST_WORKING_DIR` is mounted (matches the right side of `-v`).
`IMAGE_NAME`	Full image name of the Checkpoint Conversion Tool container.
`TARGET_CHECKPOINT_DIR`	Subdirectory under `HOST_WORKING_DIR` containing the target checkpoint directory (mirrored under `DOCKER_WORKING_DIR`).
`DRAFT_CHECKPOINT_DIR`	Subdirectory under `HOST_WORKING_DIR` containing the draft checkpoint directory (mirrored under `DOCKER_WORKING_DIR`).

Important: Both draft and target checkpoints must be converted before validation. If you are using a SambaStack-provided target checkpoint, you do not need to convert it–all SambaStack-provided checkpoints are already in the correct format.

This step is optional but strongly recommended to avoid runtime errors.

Upload your custom draft checkpoint

If your cluster uses NFS mounted storage, make sure your custom checkpoint(s) is made available in NFS where it is readable by your cluster.If you are using Google Cloud Storage for your cluster, upload the converted draft checkpoint directory to your GCS bucket and make sure it is readable by your SambaStack service account.

gcloud storage cp -r <LOCAL_DRAFT_CHECKPOINT_DIR> gs://<BUCKET_NAME>/<PATH>/

For detailed upload instructions and GCS permission configuration, see Deploying custom checkpoints → Upload your custom checkpoint.

Reference your draft checkpoint

Make the converted draft checkpoint servable using one of the two approaches described in Deploying custom checkpoints:

Override the checkpoint of the existing draft model. Set checkpointOverrides on the draft model’s entry in spec.modelConfigs. The draft keeps its existing name, so specDecodingPairs needs no change. This is the simpler option and is shown in the next step.
Register a new Model resource. Use this to serve the draft checkpoint under its own name. You must then update the draft value in specDecodingPairs to reference the new resource name.

The draft model is never addressed directly by clients, so a new serving name is rarely needed. Prefer overriding the checkpoint unless you specifically want the draft addressable under its own name.

Update your ModelBundle

Once your draft checkpoint is uploaded, point the draft model’s entry at it.

Before: using the SambaNova-provided draft

spec:
  modelConfigs:
    - model: meta-llama-3-3-70b-instruct:1
      profile: llama-3p1-70b-sd
    - model: meta-llama-3-2-1b-instruct:1
      profile: llama-3p1-1b
      modelSettings:
        routable: false
  specDecodingPairs:
    - draft: meta-llama-3-2-1b-instruct
      target: meta-llama-3-3-70b-instruct

After: using a custom draft checkpoint

Add checkpointOverrides to the draft model’s modelSettings. Nothing else changes, including specDecodingPairs:

spec:
  modelConfigs:
    - model: meta-llama-3-3-70b-instruct:1
      profile: llama-3p1-70b-sd
    - model: meta-llama-3-2-1b-instruct:1
      profile: llama-3p1-1b
      modelSettings:
        routable: false
        checkpointOverrides:
          checkpoint:
            source: gs://<BUCKET_NAME>/path/to/custom/draft/checkpoint
            checkpoint_status: stable
  specDecodingPairs:
    - draft: meta-llama-3-2-1b-instruct
      target: meta-llama-3-3-70b-instruct

A checkpoint source can be either a Google Cloud Storage path (gs://) or an NFS path (nfs://).

To customize the target checkpoint as well, add checkpointOverrides to the target model’s entry in the same way.

Apply the bundle

kubectl apply -f <modelbundle-file>.yaml

Important notes

You do not need to call the draft model directly in your inference API requests.
When you send requests to the target model, SambaStack automatically runs speculative decoding using the paired draft model.
Both models must be present in spec.modelConfigs for speculative decoding to work.
For the full custom-checkpoint workflow, see Deploying custom checkpoints.

Verify deployment

After applying the bundle and deploying it, verify your deployment:

# Check bundle status
kubectl get modelbundles

# Check bundle details, including the legalizer result
kubectl describe modelbundle <your-bundle-name>

# Check the deployment
kubectl get modeldeployments

# Verify models are registered
kubectl get models

Test by sending an inference request to your target model. SambaStack automatically uses the draft model for speculative decoding when handling the request. For examples of how to send inference requests using the SambaNova SDK, OpenAI-compatible libraries, or CURL, see the Quickstart Guide for Developers.

Troubleshooting

Issue	Possible Cause	Solution
Validation fails	Mismatched vocabulary sizes	Ensure draft and target use compatible tokenizers
Low acceptance rates	Poor draft-target alignment	Use a draft model more closely aligned with target
Performance degradation	Long inputs, short outputs	Consider disabling speculative decoding for this workload
Deployment fails	Draft model missing from the bundle	Ensure both the target and draft models appear in `spec.modelConfigs`
Model not found	Model name mismatch	Verify `modelConfigs[].model` matches the `Model` resource name

Checkpoint Conversion Tool – Convert checkpoints to SambaNova format
Deploying custom checkpoints – Deploy custom target checkpoints
Deploying models and bundles – Model, profile, bundle, and deployment concepts

Getting Started

Service Administration

Hardware Administration

Reference Architecture

Resources

Deploying with speculative decoding

What is speculative decoding

When NOT to use speculative decoding

Using custom draft checkpoints for speculative decoding

Workflow overview

Steps to deploy a custom draft checkpoint

Confirm your deployment uses speculative decoding

Speculative decoding fields

Convert your custom draft checkpoint

Validate draft–target checkpoint compatibility

Command template

Parameters

Host-level variables

Upload your custom draft checkpoint

Reference your draft checkpoint

Update your ModelBundle

Before: using the SambaNova-provided draft

After: using a custom draft checkpoint

Apply the bundle

Important notes

Verify deployment

Troubleshooting

​What is speculative decoding

​When NOT to use speculative decoding

​Using custom draft checkpoints for speculative decoding

​Workflow overview

​Steps to deploy a custom draft checkpoint

Confirm your deployment uses speculative decoding

​Speculative decoding fields

Convert your custom draft checkpoint

Validate draft–target checkpoint compatibility

​Command template

​Parameters

​Host-level variables

Upload your custom draft checkpoint

Reference your draft checkpoint

Update your ModelBundle

​Before: using the SambaNova-provided draft

​After: using a custom draft checkpoint

​Apply the bundle

​Important notes

​Verify deployment

​Troubleshooting

​Related pages

What is speculative decoding

When NOT to use speculative decoding

Using custom draft checkpoints for speculative decoding

Workflow overview

Steps to deploy a custom draft checkpoint

Speculative decoding fields

Command template

Parameters

Host-level variables

Before: using the SambaNova-provided draft

After: using a custom draft checkpoint

Apply the bundle

Important notes

Verify deployment

Troubleshooting

Related pages