> ## Documentation Index
> Fetch the complete documentation index at: https://sambanova-systems.mintlify.site/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Deploying with speculative decoding
SambaStack allows you to use custom draft models with both user-provided checkpoints and SambaNova-provided checkpoints to improve inference performance in certain deployment scenarios. These draft models enable speculative decoding, which can reduce latency and increase throughput when configured appropriately.
This page explains how to deploy models using speculative decoding. It covers when speculative decoding is appropriate, how to validate compatibility between draft and target models, and how to configure custom draft checkpoints.
If you are unfamiliar with models, model profiles, model bundles, or model deployment, see the [Deploying models and bundles](/docs/en/v2.0.2/sambastack/service-administration/model-deployment/deploying-model-bundles) page.
## What is speculative decoding
Speculative decoding is an inference acceleration technique that pairs a fast, lightweight **"draft" model** with a larger, higher-quality **"target" model**:
* The draft model proposes multiple next tokens
* The target model quickly verifies or rejects them in parallel
* This approach significantly speeds up inference decode time, especially when the draft model is closely "aligned" to the target model
**Key benefit:** Speculative decoding accelerates inference without affecting the output distributions (and thereby accuracy) of the target model.
### When NOT to use speculative decoding
Speculative decoding is most effective when the draft model can predict a meaningful portion of the target model's next tokens. In some situations, however, it can provide little benefit–or even reduce overall performance:
| Scenario | Why it hurts performance |
| --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Long inputs with short outputs** | Both the draft and target models must process the entire input before generation begins. The added overhead of running the draft model can outweigh any speedup. |
| **Very large gaps between model sizes** | When the draft model is much smaller (or otherwise poorly aligned) compared to the target model, acceptance rates tend to drop. Low acceptance rates mean more rejected tokens, more corrective forward passes on the target model, and ultimately decreased performance. |
## Using custom draft checkpoints for speculative decoding
Some model profiles in SambaStack support **speculative decoding**. When a target model uses speculative decoding, both the draft and target models must be deployed together, and the checkpoints for either model may be customized.
This section walks through the full workflow for deploying a **custom draft checkpoint**, including:
* How to identify which profiles and bundles use speculative decoding
* How to validate draft–target compatibility
* How to update your `ModelBundle`
### Workflow overview
Deploying a custom draft checkpoint for speculative decoding follows these high-level steps:
| Step | Action |
| ---- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| 1 | **Confirm** your deployment uses speculative decoding by checking the model profile and the bundle's `specDecodingPairs` |
| 2 | **Convert** your custom draft checkpoint into the SambaNova-compatible format |
| 3 | **Validate** compatibility between your draft and target checkpoints (recommended) |
| 4 | **Upload** the converted draft checkpoint to your GCS bucket, or place it on an NFS location that is accessible and readable by your cluster. |
| 5 | **Reference** your draft checkpoint, either by overriding the existing draft model's checkpoint or by registering a new `Model` |
| 6 | **Update** your `ModelBundle` to reference the custom draft checkpoint |
## Steps to deploy a custom draft checkpoint
Speculative decoding requires a target model whose `ModelProfile` supports it, together with a draft model deployed alongside it. A profile supports speculative decoding when its `spec.features` includes `speculative_decoding`:
```bash theme={}
kubectl -n get modelprofiles -o json
```
The pairing itself is declared in the `ModelBundle` through `spec.specDecodingPairs`. In the example below, `meta-llama-3-3-70b-instruct` is the target and `meta-llama-3-2-1b-instruct` is the draft. A third model, `gemma-4-31b-it`, is bundled alongside them and does not use speculative decoding.
```yaml theme={}
apiVersion: sambanova.ai/v1alpha1
kind: ModelBundle
metadata:
name: gemma4-31-llama70-sd-1b
spec:
modelConfigs:
- model: gemma-4-31b-it:1
profile: gemma4-31b-dyt
- model: meta-llama-3-3-70b-instruct:1
profile: llama-3p1-70b-sd
- model: meta-llama-3-2-1b-instruct:1
profile: llama-3p1-1b
modelSettings:
routable: false
specDecodingPairs:
- draft: meta-llama-3-2-1b-instruct
target: meta-llama-3-3-70b-instruct
```
**In the above example:**
* `meta-llama-3-3-70b-instruct` is the **target model**. It uses the `llama-3p1-70b-sd` profile, whose PEFs are compiled for speculative decoding.
* `meta-llama-3-2-1b-instruct` is the **draft model**. It is marked `routable: false`, so clients cannot address it directly.
* `spec.specDecodingPairs` declares the pairing between them.
* `gemma-4-31b-it` is unrelated to the pairing and is served normally.
Note the following:
* Both the target and the draft model must appear as entries in `spec.modelConfigs`.
* The draft model should be non-routable, since requests are sent to the target model.
* Entries in `specDecodingPairs` reference models by their bare `Model` resource name, without an architecture or version suffix.
### Speculative decoding fields
| Field | Required | Description |
| --------- | -------- | --------------------------------------------------------------------------------------------------------------------- |
| `target` | Yes | Resource name of the target model, specified without an architecture or version suffix. |
| `draft` | Yes | Resource name of the draft model, specified without an architecture or version suffix. |
| `experts` | No | Restricts the pairing to specific experts. Omit this field to apply the pairing to all of the target model's experts. |
If you are replacing the draft model's checkpoint with your own fine-tuned or custom version, convert it into the SambaNova-compatible format using the **Checkpoint Conversion Tool**.
1. If you have not already set up the tool, follow the [**Download and set up**](/docs/en/v2.0.2/sambastack/service-administration/model-deployment/checkpoint-conversion-tool#download-and-set-up) instructions on the Checkpoint Conversion Tool documentation page.
2. Then convert the draft checkpoint using the steps in [**Convert and validate checkpoint**](/docs/en/v2.0.2/sambastack/service-administration/model-deployment/checkpoint-conversion-tool#convert-and-validate-checkpoint).
**SambaStack-provided checkpoints** are already in the correct format. If you are using a SambaStack-provided draft checkpoint, skip this step.
When deploying custom checkpoints with speculative decoding, verify that a draft checkpoint is compatible with the target checkpoint. This helps prevent unexpected errors during inference.
The **Checkpoint Conversion Tool** provides a built-in utility that checks draft–target compatibility. Use the validation utility to confirm:
* The draft and target models align structurally
* Their tokenizers are compatible
* Speculative decoding can run safely with this pair
### Command template
```bash theme={}
docker run -v $HOST_WORKING_DIR:$DOCKER_WORKING_DIR --rm -it \
--platform linux/amd64 \
$IMAGE_NAME \
validate-sd \
--target_model $TARGET_MODEL \
--target_checkpoint_path "$DOCKER_WORKING_DIR/$TARGET_CHECKPOINT_DIR" \
--draft_model $DRAFT_MODEL \
--draft_checkpoint_path "$DOCKER_WORKING_DIR/$DRAFT_CHECKPOINT_DIR" \
--server $SERVER \
--cache_location $CACHE_LOCATION
```
### Parameters
These are the primary input flags to the `validate-sd` command:
| Flag / Variable | Type | Description |
| ------------------------------------- | ----- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--target_model` / `TARGET_MODEL` | `str` | The target model family to validate against (e.g., `"llama3-70b"`). This should match the model family used in your target deployment. |
| `--target_checkpoint_path` | `str` | Path **inside the container** to the directory containing the converted **target** checkpoint. Typically `$DOCKER_WORKING_DIR/` mounted from `HOST_WORKING_DIR`. |
| `--draft_model` / `DRAFT_MODEL` | `str` | The draft model family to validate (e.g., `"llama3-8b"`). This should match the model family used as the draft model in your speculative decoding configuration. |
| `--draft_checkpoint_path` | `str` | Path **inside the container** to the directory containing the converted **draft** checkpoint. Typically `$DOCKER_WORKING_DIR/` mounted from `HOST_WORKING_DIR`. |
| `--server` / `SERVER` | `str` | Source of serving metadata. Can be `embedded`, a local path, or a remote URL such as `https://api.sambanova.ai/`. Typically this is the base endpoint URL for your SambaStack instance. |
| `--cache_location` / `CACHE_LOCATION` | `str` | Location (inside the container) where serving metadata/configs are stored. Usually a subdirectory of `$DOCKER_WORKING_DIR` and visible on the host under `$HOST_WORKING_DIR/$CACHE_LOCATION`. |
### Host-level variables
In addition to the flags above, you will typically set the following environment variables for the Docker command:
| Variable | Description |
| ----------------------- | --------------------------------------------------------------------------------------------------------------------------- |
| `HOST_WORKING_DIR` | Directory on the **host** that contains your converted target and draft checkpoints. Must be writable. |
| `DOCKER_WORKING_DIR` | Directory **inside the container** where `HOST_WORKING_DIR` is mounted (matches the right side of `-v`). |
| `IMAGE_NAME` | Full image name of the Checkpoint Conversion Tool container. |
| `TARGET_CHECKPOINT_DIR` | Subdirectory under `HOST_WORKING_DIR` containing the **target** checkpoint directory (mirrored under `DOCKER_WORKING_DIR`). |
| `DRAFT_CHECKPOINT_DIR` | Subdirectory under `HOST_WORKING_DIR` containing the **draft** checkpoint directory (mirrored under `DOCKER_WORKING_DIR`). |
**Important:** Both draft **and** target checkpoints must be converted before validation. If you are using a SambaStack-provided target checkpoint, you do **not** need to convert it–all SambaStack-provided checkpoints are already in the correct format.
This step is optional but strongly recommended to avoid runtime errors.
**If your cluster uses NFS mounted storage,** make sure your custom checkpoint(s) is made available in NFS where it is readable by your cluster.
**If you are using Google Cloud Storage for your cluster,** upload the converted draft checkpoint directory to your GCS bucket and make sure it is readable by your SambaStack service account.
```bash theme={}
gcloud storage cp -r gs:////
```
For detailed upload instructions and GCS permission configuration, see [**Deploying custom checkpoints → Upload your custom checkpoint**](/docs/en/v2.0.2/sambastack/service-administration/model-deployment/deploy-custom-checkpoints#upload-your-converted-checkpoint).
Make the converted draft checkpoint servable using one of the two approaches described in [Deploying custom checkpoints](/docs/en/v2.0.2/sambastack/service-administration/model-deployment/deploy-custom-checkpoints#reference-your-checkpoint):
* **Override the checkpoint of the existing draft model.** Set `checkpointOverrides` on the draft model's entry in `spec.modelConfigs`. The draft keeps its existing name, so `specDecodingPairs` needs no change. This is the simpler option and is shown in the next step.
* **Register a new `Model` resource.** Use this to serve the draft checkpoint under its own name. You must then update the `draft` value in `specDecodingPairs` to reference the new resource name.
The draft model is never addressed directly by clients, so a new serving name is rarely needed. Prefer overriding the checkpoint unless you specifically want the draft addressable under its own name.
Once your draft checkpoint is uploaded, point the draft model's entry at it.
### Before: using the SambaNova-provided draft
```yaml theme={}
spec:
modelConfigs:
- model: meta-llama-3-3-70b-instruct:1
profile: llama-3p1-70b-sd
- model: meta-llama-3-2-1b-instruct:1
profile: llama-3p1-1b
modelSettings:
routable: false
specDecodingPairs:
- draft: meta-llama-3-2-1b-instruct
target: meta-llama-3-3-70b-instruct
```
### After: using a custom draft checkpoint
Add `checkpointOverrides` to the draft model's `modelSettings`. Nothing else changes, including `specDecodingPairs`:
```yaml theme={}
spec:
modelConfigs:
- model: meta-llama-3-3-70b-instruct:1
profile: llama-3p1-70b-sd
- model: meta-llama-3-2-1b-instruct:1
profile: llama-3p1-1b
modelSettings:
routable: false
checkpointOverrides:
checkpoint:
source: gs:///path/to/custom/draft/checkpoint
checkpoint_status: stable
specDecodingPairs:
- draft: meta-llama-3-2-1b-instruct
target: meta-llama-3-3-70b-instruct
```
A checkpoint `source` can be either a Google Cloud Storage path (`gs://`) or an NFS path (`nfs://`).
To customize the **target** checkpoint as well, add `checkpointOverrides` to the target model's entry in the same way.
### Apply the bundle
```bash theme={}
kubectl apply -f .yaml
```
### Important notes
* You **do not** need to call the draft model directly in your inference API requests.
* When you send requests to the target model, SambaStack automatically runs speculative decoding using the paired draft model.
* Both models must be present in `spec.modelConfigs` for speculative decoding to work.
* For the full custom-checkpoint workflow, see [Deploying custom checkpoints](/docs/en/v2.0.2/sambastack/service-administration/model-deployment/deploy-custom-checkpoints).
## Verify deployment
After applying the bundle and deploying it, verify your deployment:
```bash theme={}
# Check bundle status
kubectl get modelbundles
# Check bundle details, including the legalizer result
kubectl describe modelbundle
# Check the deployment
kubectl get modeldeployments
# Verify models are registered
kubectl get models
```
Test by sending an inference request to your **target model**. SambaStack automatically uses the draft model for speculative decoding when handling the request.
For examples of how to send inference requests using the SambaNova SDK, OpenAI-compatible libraries, or CURL, see the [Quickstart Guide for Developers](/docs/en/get-started/quickstart).
## Troubleshooting
| Issue | Possible Cause | Solution |
| ----------------------- | ----------------------------------- | --------------------------------------------------------------------- |
| Validation fails | Mismatched vocabulary sizes | Ensure draft and target use compatible tokenizers |
| Low acceptance rates | Poor draft-target alignment | Use a draft model more closely aligned with target |
| Performance degradation | Long inputs, short outputs | Consider disabling speculative decoding for this workload |
| Deployment fails | Draft model missing from the bundle | Ensure both the target and draft models appear in `spec.modelConfigs` |
| Model not found | Model name mismatch | Verify `modelConfigs[].model` matches the `Model` resource name |
## Related pages
* [Checkpoint Conversion Tool](/docs/en/v2.0.2/sambastack/service-administration/model-deployment/checkpoint-conversion-tool) – Convert checkpoints to SambaNova format
* [Deploying custom checkpoints](/docs/en/v2.0.2/sambastack/service-administration/model-deployment/deploy-custom-checkpoints) – Deploy custom target checkpoints
* [Deploying models and bundles](/docs/en/v2.0.2/sambastack/service-administration/model-deployment/deploying-model-bundles) – Model, profile, bundle, and deployment concepts