> ## Documentation Index > Fetch the complete documentation index at: https://sambanova-systems.mintlify.site/docs/llms.txt > Use this file to discover all available pages before exploring further. # Deploying custom checkpoints In SambaStack, you can deploy your own custom or fine-tuned checkpoints for inference in the same manner as [deploying standard model offerings](/docs/en/sambastack/service-administration/model-deployment/deploying-models-and-bundles/deploy-a-bundle), with a few additional steps to prepare your checkpoint for use in the platform. Once prepared and deployed, custom checkpoints behave just like any other checkpoint you deploy on SambaStack. For the field reference of the resources you author here, see [Model](/docs/en/sambastack/service-administration/model-deployment/custom-resources/model) and [PEF](/docs/en/sambastack/service-administration/model-deployment/custom-resources/pef) under **Custom resources**. ## Overview Deploying a custom checkpoint involves four high-level actions: 1. **Convert your checkpoint** into a SambaNova-compatible format using the [Checkpoint Conversion Tool](/docs/en/sambastack/service-administration/model-deployment/checkpoint-conversion-tool). 2. **Upload your converted checkpoint** to your private Google Cloud Storage bucket configured with read permissions granted to your SambaNova-provided service account OR make it available in NFS such that it is readable by your cluster. 3. **Reference your checkpoint**, either by overriding the checkpoint used by an existing model or by registering a new `Model` resource for it. 4. **Deploy it** by pairing the model with a compatible `ModelProfile` in a `ModelDeployment` or a `ModelBundle`. Before starting this workflow, ensure you have completed the checkpoint conversion process. See the [Checkpoint Conversion Tool](/docs/en/sambastack/service-administration/model-deployment/checkpoint-conversion-tool) page for instructions. ## Prerequisites Before deploying a custom checkpoint, ensure you have: * A converted checkpoint in SambaNova-compatible format (see [Checkpoint Conversion Tool](/docs/en/sambastack/service-administration/model-deployment/checkpoint-conversion-tool)) * Your NFS mounted storage or access to a Google Cloud Storage (GCS) bucket * Your SambaNova-provided service account JSON file * `kubectl` configured with access to your SambaStack cluster * Familiarity with the [model deployment resources](/docs/en/sambastack/service-administration/model-deployment/deploying-models-and-bundles/overview), including model profiles and model bundles ## Supported models for custom checkpoints Custom checkpoint deployment is supported for a growing set of base models in SambaStack. See the [Supported Models and Bundles](/docs/en/sambastack/service-administration/model-deployment/supported-models-and-bundles) table to find models that support custom checkpoints. ## Steps to deploy a custom checkpoint Custom or fine-tuned checkpoints must be converted into a format optimized for SambaNova's SN40L hardware before they can be deployed. SambaNova provides a **Checkpoint Conversion Tool**, delivered as a Docker container that you can run locally. The tool generates converted checkpoint artifacts that can then be uploaded and deployed for inference on SambaStack. To begin, follow the instructions in the [**Download and set up**](/docs/en/sambastack/service-administration/model-deployment/checkpoint-conversion-tool#download-and-set-up) section of the Checkpoint Conversion Tool documentation. Setup is complete once you have downloaded the conversion tool container and synced the model metadata with your specific SambaStack instance. After setup, use the steps described in the [**Convert and validate checkpoint**](/docs/en/sambastack/service-administration/model-deployment/checkpoint-conversion-tool#convert-and-validate-checkpoint) section of the Checkpoint Conversion Tool documentation to convert your custom checkpoint into the SambaNova-compatible format. **You can skip this section if you have NFS mounted to your cluster.** SambaStack uses Google Cloud Storage (GCS) to store checkpoints and other SambaStack artifacts. For custom checkpoints, you'll store the converted checkpoint artifacts in **your own** GCS bucket. To make these artifacts available to SambaStack during deployment, your SambaNova-provided service account needs read access to your bucket. This is a one-time setup step. After permissions are in place, you can upload any number of custom checkpoints to your bucket and use them directly in your deployments. ### Identifying your service account Your service account information is provided as a JSON file. Locate the `client_email` field - this is the identity that needs read access to your bucket. For example: ```json theme={} { "type": "service_account", "project_id": "example-project-id", "private_key_id": "example-private-key-id", "private_key": "-----BEGIN PRIVATE KEY-----\n\n-----END PRIVATE KEY-----\n", "client_email": "ss-artifacts-reader@example-project-id.iam.gserviceaccount.com", "client_id": "12345678901234567890", "auth_uri": "https://accounts.google.com/o/oauth2/auth", "token_uri": "https://oauth2.googleapis.com/token", "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/ss-artifacts-reader%40example-project-id.iam.gserviceaccount.com", "universe_domain": "googleapis.com" } ``` ### Granting Storage Object Viewer role To allow SambaStack to access your custom checkpoints, grant the service account the **Storage Object Viewer** role on your bucket. This provides read-only access to objects without allowing writes or modifications. **Using the Google Cloud Console:** 1. Open the **Google Cloud Console**. 2. Navigate to **Storage → Buckets**, and select the bucket you plan to use. 3. Go to the **Permissions** tab. 4. Click **+ Add principal**. 5. In the *New principals* field, enter your service account's `client_email`. 6. In the *Role* dropdown, choose: **Cloud Storage → Storage Object Viewer** 7. Click **Save**. **Using the gcloud CLI:** Before running the commands below, identify: * `` – the name of your GCS bucket * `` – the `client_email` value from your service account JSON * `` – the Google Cloud project that owns the bucket To grant the **Storage Object Viewer** role at the bucket level: ```bash theme={} gcloud storage buckets add-iam-policy-binding gs:// \ --member="serviceAccount:" \ --role="roles/storage.objectViewer" \ --project= ``` To verify that the role was successfully applied: ```bash theme={} gcloud storage buckets get-iam-policy gs:// \ --project= ``` You should see an entry resembling: ```yaml theme={} bindings: - members: - serviceAccount: role: roles/storage.objectViewer ``` For additional guidance, see Google's IAM documentation: * [Google Cloud: Granting, changing, and revoking access to resources](https://cloud.google.com/iam/docs/granting-changing-revoking-access) * [Google Cloud Storage IAM roles](https://cloud.google.com/storage/docs/access-control/iam-roles) ### NFS If you have NFS mounted to your cluster, verify that the converted checkpoint is moved to NFS and accessible by your cluster. ### Google Cloud Storage If you are using Google Cloud Storage (GCS), after converting the checkpoint, upload the directory containing the converted checkpoint files to your GCS bucket. This step may take a while depending on the size of your checkpoint. **Using the Google Cloud Console:** 1. Open the **Google Cloud Console**. 2. Navigate to **Storage → Buckets** and select the bucket you've configured for custom checkpoints. 3. Click **Upload folder** (or **Upload files**, depending on your structure). 4. Select the directory containing your converted checkpoint artifacts. 5. Wait for the upload to complete; the structure should remain intact. **Using the gcloud CLI:** You can upload the entire converted checkpoint directory recursively with: ```bash theme={} gcloud storage cp -r gs://// ``` There are two ways to make your checkpoint servable, depending on whether you want to keep the existing model name or serve the checkpoint under a new one. ### Option 1: Override the checkpoint of an existing model Use this option to reuse an existing model's name and serve your checkpoint in place of the one that model normally uses. No new `Model` resource is required. Set `checkpointOverrides` on the model configuration in your `ModelBundle` or `ModelDeployment`: ```yaml theme={} spec: modelConfigs: - model: meta-llama-3-1-8b-instruct:1 profile: llama-3p1-8b modelSettings: checkpointOverrides: checkpoint: source: gs:///path/to/converted/checkpoint checkpoint_status: stable tool_support: false ``` Requests continue to use the existing model's serving name. This is the recommended approach for serving a fine-tuned variant of a supported model. ### Option 2: Register a new Model resource Use this option to serve the checkpoint under its **own** model name, for example a checkpoint fine-tuned from another model that you want addressed separately in the inference API. This requires creating a new `Model` resource. Set `spec.checkpoints.` using an architecture key compatible with the profile you intend to pair it with, set `source` to the location of your converted checkpoint, and set `spec.tokenizer.path` to the tokenizer of the base model the checkpoint was derived from. ```yaml theme={} apiVersion: sambanova.ai/v1alpha1 kind: Model metadata: name: my-custom-llama-3-1-8b # must be a valid Kubernetes resource name spec: name: my-custom-llama-3-1-8b # serving name used in the inference API owner: jane@doe.ai public: true aliases: - My-Custom-Llama3.1-8B checkpoints: llama3: versions: "1": source: gs:///path/to/converted/checkpoint checkpoint_status: stable tool_support: false tokenizer: path: ./Meta-Llama-3.1-8B-Instruct_tokenizer metadata: capabilities: - text ``` The architecture key in this example, `llama3`, is illustrative. Use a key that matches the `model_arch` of the profile you intend to use. For the full field reference, see [Model](/docs/en/sambastack/service-administration/model-deployment/custom-resources/model). The `tokenizer` field is used only for checking inputs to calculate sequence length requirements prior to generation time. Set it to the tokenizer of the base model your checkpoint was fine-tuned from. Apply the resource: ```bash theme={} kubectl apply -f .yaml ``` ```bash theme={} kubectl -n apply -f .yaml ``` Pair the model with a `ModelProfile` whose `model_arch` is compatible with your checkpoint's architecture, then deploy it either directly or through a bundle. Compatibility between a checkpoint and a profile is determined by `model_arch`, and the operator does not verify it for you. Select a profile whose architecture matches the checkpoint you are serving. A mismatch results in inference errors. To find a compatible profile, list the profiles in your cluster and note the `model_arch` of each: ```bash theme={} kubectl get modelprofiles ``` ```bash theme={} kubectl -n get modelprofiles.sambanova.ai ``` Then deploy. For a single model, inline it in a `ModelDeployment`: ```yaml theme={} apiVersion: sambanova.ai/v1alpha1 kind: ModelDeployment metadata: name: md-my-custom-llama-3-1-8b spec: models: modelConfigs: - model: my-custom-llama-3-1-8b:1 profile: llama-3p1-8b groups: - minReplicas: 1 name: default qosList: - free owner: jane@doe.ai secretNames: - sambanova-artifact-reader ``` Apply it: ```bash theme={} kubectl apply -f .yaml ``` ```bash theme={} kubectl -n apply -f .yaml ``` For the full deployment options, including bundling several models together, see [Deploy a bundle](/docs/en/sambastack/service-administration/model-deployment/deploying-models-and-bundles/deploy-a-bundle). After your deployment is running, use the serving name you defined (for example, `my-custom-llama-3-1-8b`) in your inference API requests. ## Add a custom model architecture If the model architecture is not yet supported, no compiled executable and no profile exist for it, so the full set of resources must be authored in the following order: | Order | Resource | Purpose | | ----- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | 1 | `Pef` | Registers a PEF compiled for the new architecture, including its `spec.metadata.model_arch`, batch size, maximum sequence length, and feature support flags. | | 2 | `ModelProfile` | Lists the new PEFs in `spec.pefs`, surfaces the new `model_arch` value, and defines the default batching support for the architecture. | | 3 | `Model` | Holds the checkpoint sources under the same architecture key as the PEFs, together with the tokenizer path. | | 4 | `ModelDeployment` | Deploys the model, either directly or through a `ModelBundle`. | A new PEF must be compiled for the architecture, and a `Pef` resource must be created to register it. Compiling a PEF is not covered on this page. Use the resources for the closest supported model as a reference for the values in each resource: ```bash theme={} kubectl get pef -o yaml kubectl get modelprofile -o yaml kubectl get model -o yaml ``` ```bash theme={} kubectl -n get pef.sambanova.ai -o yaml kubectl -n get modelprofile.sambanova.ai -o yaml kubectl -n get model.sambanova.ai -o yaml ``` ## Verifying your deployment After applying the deployment, verify that your custom checkpoint deployment is successful: 1. **Check deployment status:** ```bash theme={} kubectl get modeldeployments kubectl describe modeldeployment ``` 2. **Verify the model is available:** ```bash theme={} kubectl get models ``` This lists Kubernetes resource names (`metadata.name`). The name your request must use is the serving name you set in `spec.name`, which is often different. 3. **Test with a sample inference request** using the serving name from `spec.name`. See the [Quickstart Guide for Developers](/docs/en/get-started/quickstart) for example inference requests using the SambaNova SDK, OpenAI-compatible libraries, or CURL. ## Troubleshooting ### Common issues | Issue | Possible Cause | Solution | | -------------------------------------- | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | | Deployment fails with permission error | Service account lacks read access to GCS bucket | Verify Storage Object Viewer role is granted (see Step 2) | | Model not found in API requests | Model name mismatch, or the model is not routable | Ensure requests use `spec.name` from the `Model` resource, and that `modelSettings.routable` is not set to `false` | | Checkpoint files not found | Incorrect checkpoint path | Verify the `source` path matches your uploaded checkpoint location | | Inference errors | Checkpoint incompatible with the profile | Ensure the profile's `model_arch` is compatible with your checkpoint's architecture | ### Verifying GCS access If you suspect permission issues, verify that your service account can access the checkpoint: ```bash theme={} gcloud auth activate-service-account --key-file= gsutil ls gs://// ``` ## Next steps * For the resources you author, see [Model](/docs/en/sambastack/service-administration/model-deployment/custom-resources/model), [PEF](/docs/en/sambastack/service-administration/model-deployment/custom-resources/pef), and the [deployment overview](/docs/en/sambastack/service-administration/model-deployment/deploying-models-and-bundles/overview) * To deploy custom checkpoints with speculative decoding, see [Deploying with speculative decoding](/docs/en/sambastack/service-administration/performance/deploy-with-speculative-decoding) * For monitoring and observability, see [SambaStack Monitoring](/docs/en/sambastack/reference-architecture/observability/overview) * If your custom checkpoint uses a different chat template or tool-call output format than the base model, see [Custom chat templates and output parsing](/docs/en/build/chat-templates) in the Developer Guide for how to handle prompt formatting and parsing on the client side.