> ## Documentation Index
> Fetch the complete documentation index at: https://sambanova-systems.mintlify.site/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Deploying custom checkpoints
> Convert, upload, and serve your own custom or fine-tuned checkpoints on SambaStack, either by overriding the checkpoint of an existing model or by registering a new Model resource.
In SambaStack, you can deploy your own custom or fine-tuned checkpoints for inference in the same manner as [deploying standard model offerings](/docs/en/v2.0.2/sambastack/service-administration/model-deployment/deploying-model-bundles), with a few additional steps to prepare your checkpoint for use in the platform. Once prepared and deployed, custom checkpoints behave just like any other checkpoint you deploy on SambaStack.
This page also documents the `Model` and `Pef` resources, which you author when bringing your own checkpoint or model architecture.
## Overview
Deploying a custom checkpoint involves four high-level actions:
1. **Convert your checkpoint** into a SambaNova-compatible format using the [Checkpoint Conversion Tool](/docs/en/v2.0.2/sambastack/service-administration/model-deployment/checkpoint-conversion-tool).
2. **Upload your converted checkpoint** to your private Google Cloud Storage bucket configured with read permissions granted to your SambaNova-provided service account OR make it available in NFS such that it is readable by your cluster.
3. **Reference your checkpoint**, either by overriding the checkpoint used by an existing model or by registering a new `Model` resource for it.
4. **Deploy it** by pairing the model with a compatible `ModelProfile` in a `ModelDeployment` or a `ModelBundle`.
Before starting this workflow, ensure you have completed the checkpoint conversion process. See the [Checkpoint Conversion Tool](/docs/en/v2.0.2/sambastack/service-administration/model-deployment/checkpoint-conversion-tool) page for instructions.
## Prerequisites
Before deploying a custom checkpoint, ensure you have:
* A converted checkpoint in SambaNova-compatible format (see [Checkpoint Conversion Tool](/docs/en/v2.0.2/sambastack/service-administration/model-deployment/checkpoint-conversion-tool))
* Your NFS mounted storage or access to a Google Cloud Storage (GCS) bucket
* Your SambaNova-provided service account JSON file
* `kubectl` configured with access to your SambaStack cluster
* Familiarity with [deploying models and bundles](/docs/en/v2.0.2/sambastack/service-administration/model-deployment/deploying-model-bundles), including model profiles and model bundles
## Supported models for custom checkpoints
Custom checkpoint deployment is supported for a growing set of base models in SambaStack. See the [Supported Models and Bundles](/docs/en/v2.0.2/sambastack/service-administration/model-deployment/supported-models-and-bundles) table to find models that support custom checkpoints.
## Steps to deploy a custom checkpoint
Custom or fine-tuned checkpoints must be converted into a format optimized for SambaNova's SN40L hardware before they can be deployed. SambaNova provides a **Checkpoint Conversion Tool**, delivered as a Docker container that you can run locally. The tool generates converted checkpoint artifacts that can then be uploaded and deployed for inference on SambaStack.
To begin, follow the instructions in the [**Download and set up**](/docs/en/v2.0.2/sambastack/service-administration/model-deployment/checkpoint-conversion-tool#download-and-set-up) section of the Checkpoint Conversion Tool documentation. Setup is complete once you have downloaded the conversion tool container and synced the model metadata with your specific SambaStack instance.
After setup, use the steps described in the [**Convert and validate checkpoint**](/docs/en/v2.0.2/sambastack/service-administration/model-deployment/checkpoint-conversion-tool#convert-and-validate-checkpoint) section of the Checkpoint Conversion Tool documentation to convert your custom checkpoint into the SambaNova-compatible format.
**You can skip this section if you have NFS mounted to your cluster.**
SambaStack uses Google Cloud Storage (GCS) to store checkpoints and other SambaStack artifacts. For custom checkpoints, you'll store the converted checkpoint artifacts in **your own** GCS bucket. To make these artifacts available to SambaStack during deployment, your SambaNova-provided service account needs read access to your bucket.
This is a one-time setup step. After permissions are in place, you can upload any number of custom checkpoints to your bucket and use them directly in your deployments.
### Identifying your service account
Your service account information is provided as a JSON file. Locate the `client_email` field - this is the identity that needs read access to your bucket. For example:
```json theme={}
{
"type": "service_account",
"project_id": "example-project-id",
"private_key_id": "example-private-key-id",
"private_key": "-----BEGIN PRIVATE KEY-----\n\n-----END PRIVATE KEY-----\n",
"client_email": "ss-artifacts-reader@example-project-id.iam.gserviceaccount.com",
"client_id": "12345678901234567890",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/ss-artifacts-reader%40example-project-id.iam.gserviceaccount.com",
"universe_domain": "googleapis.com"
}
```
### Granting Storage Object Viewer role
To allow SambaStack to access your custom checkpoints, grant the service account the **Storage Object Viewer** role on your bucket. This provides read-only access to objects without allowing writes or modifications.
**Using the Google Cloud Console:**
1. Open the **Google Cloud Console**.
2. Navigate to **Storage → Buckets**, and select the bucket you plan to use.
3. Go to the **Permissions** tab.
4. Click **+ Add principal**.
5. In the *New principals* field, enter your service account's `client_email`.
6. In the *Role* dropdown, choose: **Cloud Storage → Storage Object Viewer**
7. Click **Save**.
**Using the gcloud CLI:**
Before running the commands below, identify:
* `` – the name of your GCS bucket
* `` – the `client_email` value from your service account JSON
* `` – the Google Cloud project that owns the bucket
To grant the **Storage Object Viewer** role at the bucket level:
```bash theme={}
gcloud storage buckets add-iam-policy-binding gs:// \
--member="serviceAccount:" \
--role="roles/storage.objectViewer" \
--project=
```
To verify that the role was successfully applied:
```bash theme={}
gcloud storage buckets get-iam-policy gs:// \
--project=
```
You should see an entry resembling:
```yaml theme={}
bindings:
- members:
- serviceAccount:
role: roles/storage.objectViewer
```
For additional guidance, see Google's IAM documentation:
* [Google Cloud: Granting, changing, and revoking access to resources](https://cloud.google.com/iam/docs/granting-changing-revoking-access)
* [Google Cloud Storage IAM roles](https://cloud.google.com/storage/docs/access-control/iam-roles)
### NFS
If you have NFS mounted to your cluster, verify that the converted checkpoint is moved to NFS and accessible by your cluster.
### Google Cloud Storage
If you are using Google Cloud Storage (GCS), after converting the checkpoint, upload the directory containing the converted checkpoint files to your GCS bucket.
This step may take a while depending on the size of your checkpoint.
**Using the Google Cloud Console:**
1. Open the **Google Cloud Console**.
2. Navigate to **Storage → Buckets** and select the bucket you've configured for custom checkpoints.
3. Click **Upload folder** (or **Upload files**, depending on your structure).
4. Select the directory containing your converted checkpoint artifacts.
5. Wait for the upload to complete; the structure should remain intact.
**Using the gcloud CLI:**
You can upload the entire converted checkpoint directory recursively with:
```bash theme={}
gcloud storage cp -r gs:////
```
There are two ways to make your checkpoint servable, depending on whether you want to keep the existing model name or serve the checkpoint under a new one.
### Option 1: Override the checkpoint of an existing model
Use this option to reuse an existing model's name and serve your checkpoint in place of the one that model normally uses. No new `Model` resource is required. Set `checkpointOverrides` on the model configuration in your `ModelBundle` or `ModelDeployment`:
```yaml theme={}
spec:
modelConfigs:
- model: meta-llama-3-1-8b-instruct:1
profile: llama-3p1-8b
modelSettings:
checkpointOverrides:
checkpoint:
source: gs:///path/to/converted/checkpoint
checkpoint_status: stable
tool_support: false
```
Requests continue to use the existing model's serving name. This is the recommended approach for serving a fine-tuned variant of a supported model.
### Option 2: Register a new Model resource
Use this option to serve the checkpoint under its **own** model name, for example a checkpoint fine-tuned from another model that you want addressed separately in the inference API. This requires creating a new `Model` resource.
Set `spec.checkpoints.` using an architecture key compatible with the profile you intend to pair it with, set `source` to the location of your converted checkpoint, and set `spec.tokenizer.path` to the tokenizer of the base model the checkpoint was derived from.
```yaml theme={}
apiVersion: sambanova.ai/v1alpha1
kind: Model
metadata:
name: my-custom-llama-3-1-8b # must be a valid Kubernetes resource name
spec:
name: my-custom-llama-3-1-8b # serving name used in the inference API
owner: jane@doe.ai
public: true
aliases:
- My-Custom-Llama3.1-8B
checkpoints:
llama3:
versions:
"1":
source: gs:///path/to/converted/checkpoint
checkpoint_status: stable
tool_support: false
tokenizer:
path: ./Meta-Llama-3.1-8B-Instruct_tokenizer
metadata:
capabilities:
- text
```
The architecture key in this example, `llama3`, is illustrative. Use a key that matches the `model_arch` of the profile you intend to use. For the full field reference, see [Model structure](#model-structure).
The `tokenizer` field is used only for checking inputs to calculate sequence length requirements prior to generation time. Set it to the tokenizer of the base model your checkpoint was fine-tuned from.
Apply the resource:
```bash theme={}
kubectl apply -f .yaml
```
```bash theme={}
kubectl -n apply -f .yaml
```
Pair the model with a `ModelProfile` whose `model_arch` is compatible with your checkpoint's architecture, then deploy it either directly or through a bundle.
Compatibility between a checkpoint and a profile is determined by `model_arch`, and the operator does not verify it for you. Select a profile whose architecture matches the checkpoint you are serving. A mismatch results in inference errors.
To find a compatible profile, list the profiles in your cluster and note the `model_arch` of each:
```bash theme={}
kubectl get modelprofiles
```
```bash theme={}
kubectl -n get modelprofiles.sambanova.ai
```
Then deploy. For a single model, inline it in a `ModelDeployment`:
```yaml theme={}
apiVersion: sambanova.ai/v1alpha1
kind: ModelDeployment
metadata:
name: md-my-custom-llama-3-1-8b
spec:
models:
modelConfigs:
- model: my-custom-llama-3-1-8b:1
profile: llama-3p1-8b
groups:
- minReplicas: 1
name: default
qosList:
- free
owner: jane@doe.ai
secretNames:
- sambanova-artifact-reader
```
Apply it:
```bash theme={}
kubectl apply -f .yaml
```
```bash theme={}
kubectl -n apply -f .yaml
```
For the full deployment options, including bundling several models together, see [Deploying models and bundles](/docs/en/v2.0.2/sambastack/service-administration/model-deployment/deploying-model-bundles).
After your deployment is running, use the serving name you defined (for example, `my-custom-llama-3-1-8b`) in your inference API requests.
## Resource reference
The following resources are the ones you author when bringing your own checkpoint or model architecture. For the profile, bundle, and deployment resources, see [Deploying models and bundles](/docs/en/v2.0.2/sambastack/service-administration/model-deployment/deploying-model-bundles).
### Model structure
A Model resource is the source of checkpoints. It holds the checkpoint versions for each architecture the model supports, keyed by architecture name, along with the tokenizer to use.
```yaml theme={}
apiVersion: sambanova.ai/v1alpha1
kind: Model
metadata:
name: deepseek-v3-2
spec:
name: DeepSeek-V3.2
owner: no-reply@sambanova.ai
checkpoints:
deepseek:
versions:
"1":
source: gs:///path/to/checkpoint
checkpoint_status: stable
tool_support: true
tokenizer:
path: ./DeepSeek-V3.2_tokenizer
metadata:
capabilities:
- text
```
| Field | Required | Description |
| ------------------------------------------------------------------------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `metadata.name` | Yes | Kubernetes resource name. This value is used in `modelConfigs[].model` references. |
| `spec.name` | Yes | Serving name used in the inference API. |
| `spec.aliases` | No | Alternative names that route to this model. |
| `spec.checkpoints.` | Yes | Checkpoints keyed by architecture. The `` key must match the `model_arch` value of the PEFs in the profile that this model is paired with. |
| `spec.checkpoints..versions..source` | Yes | Path pointing to the model checkpoint. Find available checkpoints in [Supported Models and Bundles](/docs/en/v2.0.2/sambastack/service-administration/model-deployment/supported-models-and-bundles). |
| `spec.checkpoints..versions..checkpoint_status` | Yes | Lifecycle status of this checkpoint version. See [PEF and checkpoint lifecycle status](#pef-and-checkpoint-lifecycle-status). |
| `spec.checkpoints..versions..tool_support` | No | Whether this checkpoint is compatible with tools and function calling. Defaults to `false`. |
| `spec.checkpoints..versions..vision_embedding_checkpoint` | No | Companion vision embedding checkpoint, used for multimodal models. |
| `spec.tokenizer.path` | Yes | Path to the local tokenizer to use. For a custom checkpoint, set this to the tokenizer of the base model that the checkpoint was derived from. |
| `spec.tokenizer.endpointUrls` | No | URLs to send tokenizer requests to. One URL is selected at random per request. |
| `spec.owner` | Yes | Email address of the model owner for tracking and notifications. |
| `spec.tool_support` | No | Whether the model supports using tools. |
| `spec.public` | No | Whether the model appears in the public models list. |
| `spec.metadata.capabilities` | Yes | Capabilities of the model, for example `text`, `vision`, `audio`, `embeddings`, or `reasoning`. |
| `spec.metadata` | No | Additional catalog metadata surfaced in the models API and documentation: `provider`, `license`, `overview`, `architecture`, `languages`, `category`, `text_only`, and `vocabulary_size`. |
| `spec.expertFields` | No | Defaults applied to every expert instance of this model at deployment time: `output_processor`, `postprocess_parser`, `enable_reasoning_effort`, and `tiktoken_vocab_rel_path`. Bundle-side values for the same keys take precedence. |
| `spec.price` | No | Pricing for the model: `input_tokens`, `output_tokens`, `input_cache_read`, `input_cache_write`, and `input_audio_duration_per_hour`. |
| `spec.fallback` | No | Fallback configuration, typically for multimodal support. Takes `model` and `when_missing`, both required when set. |
| `spec.api_forward_url` | No | Forwards traffic for this model to another cluster. |
The paths to checkpoints hosted by SambaNova will be provided to you by your SambaNova contact. If you have hosted your own checkpoints, you can include those paths in the `source` fields above.
### PEF structure
A Pef resource registers a compiled executable. The profile determines which PEF, sequence size, and batch size are used, so you select a profile rather than an individual PEF. Author `Pef` resources only when introducing a new model architecture.
| Field | Required | Description |
| --------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------- |
| `spec.metadata.model_arch` | Yes | Architecture of this PEF, which determines the checkpoints that the PEF is compatible with. |
| `spec.metadata.batch_size` | Yes | Batch size that the PEF was compiled for. |
| `spec.metadata.max_seq_length` | Yes | Maximum sequence length that the PEF supports. |
| `spec.metadata.seq_lengths` | No | Sequence lengths that the PEF supports. The maximum value must equal `max_seq_length`. |
| `spec.metadata.num_rdus`, `spec.metadata.rdu_arch` | No | Number of RDUs and the RDU architecture that the PEF was compiled for, for example `sn40`. |
| `spec.metadata.continuous_batching`, `spec.metadata.use_context_cache`, `spec.metadata.constrained_decoding`, `spec.metadata.is_spec_prefill` | No | Feature support flags. These values determine the `features` list of the profiles built from this PEF. |
| `spec.metadata.dynamic_dims` | No | Dynamic dimensions available in this PEF, such as `batch_size` and the sequence length dimensions. |
| `spec.versions..source` | Yes | Location to download this version of the PEF from. |
| `spec.versions..cached_path` | Yes | Path to this PEF as cached in the legalizer database. |
| `spec.versions..ckpt_sharing_uuid` | Yes | Identifier that defines checkpoint-sharing relationships. PEFs with the same identifier can share a checkpoint. |
| `spec.versions..pef_status` | Yes | Lifecycle status of this PEF version. See [PEF and checkpoint lifecycle status](#pef-and-checkpoint-lifecycle-status). |
| `spec.owner` | Yes | Owner of the PEF. |
| `spec.secretName` | Yes | Name of the single secret required in order to download the artifact. |
### PEF and checkpoint lifecycle status
SambaStack assigns a `pef_status` field to PEF CR versions and a `checkpoint_status` field to model CR checkpoint versions to indicate their support lifecycle. Understanding these statuses helps you make informed decisions when selecting PEF or checkpoint versions.
**PEF and checkpoint version status values**
Each version entry in a PEF CR includes a `pef_status` field. Model CR checkpoint versions use `checkpoint_status`. Both share the same set of values:
| Status | Description |
| ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `preview` | Not fully tested or supported. May have unknown reliability or performance issues, or limited functionality (for example, partial function calling support). Not recommended for production workloads. |
| `stable` | Fully supported and tested. |
| `deprecated` | Has known reliability or performance issues. Still available for a limited transition period (up to 3 months from the deprecation announcement) to allow migration to a stable version. |
| `removed` | No longer usable. The version entry is retained in the PEF CR or model CR for traceability and auditability, but the path may no longer exist, causing deployment to fail if referenced. |
**Example PEF CR versions with status**
```yaml theme={}
versions:
'1':
source: gs://ext-sambastack-artifacts-prod-0/path/to/pef_v1.pef
pef_status: deprecated
'2':
source: gs://ext-sambastack-artifacts-prod-0/path/to/pef_v2.pef
pef_status: stable
```
To check version statuses, run `kubectl describe pef ` or `kubectl describe model ` and review the `pef_status` or `checkpoint_status` field in the `Versions` section.
The following procedures describe the step-by-step workflow for creating and deploying custom deployments and bundles using the concepts and structures described above.
### Add a custom model architecture
If the model architecture is not yet supported, no compiled executable and no profile exist for it, so the full set of resources must be authored in the following order:
| Order | Resource | Purpose |
| ----- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| 1 | `Pef` | Registers a PEF compiled for the new architecture, including its `spec.metadata.model_arch`, batch size, maximum sequence length, and feature support flags. |
| 2 | `ModelProfile` | Lists the new PEFs in `spec.pefs`, surfaces the new `model_arch` value, and defines the default batching support for the architecture. |
| 3 | `Model` | Holds the checkpoint sources under the same architecture key as the PEFs, together with the tokenizer path. |
| 4 | `ModelDeployment` | Deploys the model, either directly or through a `ModelBundle`. |
A new PEF must be compiled for the architecture, and a `Pef` resource must be created to register it. Compiling a PEF is not covered on this page.
Use the resources for the closest supported model as a reference for the values in each resource:
```bash theme={}
kubectl get pef -o yaml
kubectl get modelprofile -o yaml
kubectl get model -o yaml
```
```bash theme={}
kubectl -n get pef.sambanova.ai -o yaml
kubectl -n get modelprofile.sambanova.ai -o yaml
kubectl -n get model.sambanova.ai -o yaml
```
## Verifying your deployment
After applying the deployment, verify that your custom checkpoint deployment is successful:
1. **Check deployment status:**
```bash theme={}
kubectl get modeldeployments
kubectl describe modeldeployment
```
2. **Verify the model is available:**
```bash theme={}
kubectl get models
```
3. **Test with a sample inference request** using your custom model name.
See the [Quickstart Guide for Developers](/docs/en/get-started/quickstart) for example inference requests using the SambaNova SDK, OpenAI-compatible libraries, or CURL.
## Troubleshooting
### Common issues
| Issue | Possible Cause | Solution |
| -------------------------------------- | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
| Deployment fails with permission error | Service account lacks read access to GCS bucket | Verify Storage Object Viewer role is granted (see Step 2) |
| Model not found in API requests | Model name mismatch, or the model is not routable | Ensure requests use `spec.name` from the `Model` resource, and that `modelSettings.routable` is not set to `false` |
| Checkpoint files not found | Incorrect checkpoint path | Verify the `source` path matches your uploaded checkpoint location |
| Inference errors | Checkpoint incompatible with the profile | Ensure the profile's `model_arch` is compatible with your checkpoint's architecture |
### Verifying GCS access
If you suspect permission issues, verify that your service account can access the checkpoint:
```bash theme={}
gcloud auth activate-service-account --key-file=
gsutil ls gs:////
```
## Next steps
* To deploy custom checkpoints with speculative decoding, see [Deploying with speculative decoding](/docs/en/v2.0.2/sambastack/service-administration/performance/deploy-with-speculative-decoding)
* For monitoring and observability, see [SambaStack Monitoring](/docs/en/v2.0.2/sambastack/reference-architecture/observability/overview)
* If your custom checkpoint uses a different chat template or tool-call output format than the base model, see [Custom chat templates and output parsing](/docs/en/build/chat-templates) in the Developer Guide for how to handle prompt formatting and parsing on the client side.