> ## Documentation Index > Fetch the complete documentation index at: https://sambanova-systems.mintlify.site/docs/llms.txt > Use this file to discover all available pages before exploring further. # Deploying custom checkpoints In SambaStack, you can deploy your own custom or fine-tuned checkpoints for inference in the same manner as [deploying standard model offerings](/en/v1.2.0/sambastack/service-administration/deploying-model-bundles), with a few additional steps to prepare your checkpoint for use in the platform. Once prepared and deployed, custom checkpoints behave just like any other checkpoint you deploy on SambaStack. ## Overview Deploying a custom checkpoint involves four high-level actions: 1. **Convert your checkpoint** into a SambaNova-compatible format using the [Checkpoint Conversion Tool](/en/v1.2.0/sambastack/service-administration/checkpoint-conversion-tool). 2. **Upload your converted checkpoint** to your private Google Cloud Storage bucket configured with read permissions granted to your SambaNova-provided service account OR make it available in NFS such that it is readable by your cluster. 3. **Register your checkpoint** by creating a Model Manifest. 4. **Reference the checkpoint in your deployment Bundle** by providing its path and specifying a name for it. Before starting this workflow, ensure you have completed the checkpoint conversion process. See the [Checkpoint Conversion Tool](/en/v1.2.0/sambastack/service-administration/checkpoint-conversion-tool) page for instructions. ## Prerequisites Before deploying a custom checkpoint, ensure you have: * A converted checkpoint in SambaNova-compatible format (see [Checkpoint Conversion Tool](/en/v1.2.0/sambastack/service-administration/checkpoint-conversion-tool)) * Your NFS mounted storage or access to a Google Cloud Storage (GCS) bucket * Your SambaNova-provided service account JSON file * `kubectl` configured with access to your SambaStack cluster * Familiarity with [model deployment](/en/v1.2.0/sambastack/service-administration/deploying-model-bundles) concepts including Bundles and Bundle Templates ## Supported models for custom checkpoints Custom checkpoint deployment is supported for a growing set of base models in SambaStack. See the [Supported Models and Bundles](/en/v1.2.0/sambastack/service-administration/supported-models-and-bundles) table to find models that support custom checkpoints. ## Steps to deploy a custom checkpoint Custom or fine-tuned checkpoints must be converted into a format optimized for SambaNova's SN40L hardware before they can be deployed. SambaNova provides a **Checkpoint Conversion Tool**, delivered as a Docker container that you can run locally. The tool generates converted checkpoint artifacts that can then be uploaded and deployed for inference on SambaStack. To begin, follow the instructions in the [**Download and set up**](/en/v1.2.0/sambastack/service-administration/checkpoint-conversion-tool#download-and-set-up) section of the Checkpoint Conversion Tool documentation. Setup is complete once you have downloaded the conversion tool container and synced the model metadata with your specific SambaStack instance. After setup, use the steps described in the [**Convert and validate checkpoint**](/en/v1.2.0/sambastack/service-administration/checkpoint-conversion-tool#convert-and-validate-checkpoint) section of the Checkpoint Conversion Tool documentation to convert your custom checkpoint into the SambaNova-compatible format. **You can skip this section if you have NFS mounted to your cluster.** SambaStack uses Google Cloud Storage (GCS) to store checkpoints and other SambaStack artifacts. For custom checkpoints, you'll store the converted checkpoint artifacts in **your own** GCS bucket. To make these artifacts available to SambaStack during deployment, your SambaNova-provided service account needs read access to your bucket. This is a one-time setup step. After permissions are in place, you can upload any number of custom checkpoints to your bucket and use them directly in your deployments. ### Identifying your service account Your service account information is provided as a JSON file. Locate the `client_email` field - this is the identity that needs read access to your bucket. For example: ```json theme={null} { "type": "service_account", "project_id": "example-project-id", "private_key_id": "example-private-key-id", "private_key": "-----BEGIN PRIVATE KEY-----\n\n-----END PRIVATE KEY-----\n", "client_email": "ss-artifacts-reader@example-project-id.iam.gserviceaccount.com", "client_id": "12345678901234567890", "auth_uri": "https://accounts.google.com/o/oauth2/auth", "token_uri": "https://oauth2.googleapis.com/token", "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/ss-artifacts-reader%40example-project-id.iam.gserviceaccount.com", "universe_domain": "googleapis.com" } ``` ### Granting Storage Object Viewer role To allow SambaStack to access your custom checkpoints, grant the service account the **Storage Object Viewer** role on your bucket. This provides read-only access to objects without allowing writes or modifications. **Using the Google Cloud Console:** 1. Open the **Google Cloud Console**. 2. Navigate to **Storage → Buckets**, and select the bucket you plan to use. 3. Go to the **Permissions** tab. 4. Click **+ Add principal**. 5. In the *New principals* field, enter your service account's `client_email`. 6. In the *Role* dropdown, choose: **Cloud Storage → Storage Object Viewer** 7. Click **Save**. **Using the gcloud CLI:** Before running the commands below, identify: * `` – the name of your GCS bucket * `` – the `client_email` value from your service account JSON * `` – the Google Cloud project that owns the bucket To grant the **Storage Object Viewer** role at the bucket level: ```bash theme={null} gcloud storage buckets add-iam-policy-binding gs:// \ --member="serviceAccount:" \ --role="roles/storage.objectViewer" \ --project= ``` To verify that the role was successfully applied: ```bash theme={null} gcloud storage buckets get-iam-policy gs:// \ --project= ``` You should see an entry resembling: ```yaml theme={null} bindings: - members: - serviceAccount: role: roles/storage.objectViewer ``` For additional guidance, see Google's IAM documentation: * [Google Cloud: Granting, changing, and revoking access to resources](https://cloud.google.com/iam/docs/granting-changing-revoking-access) * [Google Cloud Storage IAM roles](https://cloud.google.com/storage/docs/access-control/iam-roles) ### NFS If you have NFS mounted to your cluster, verify that the converted checkpoint is moved to NFS and accessible by your cluster. ### Google Cloud Storage If you are using Google Cloud Storage (GCS), after converting the checkpoint, upload the directory containing the converted checkpoint files to your GCS bucket. This step may take a while depending on the size of your checkpoint. **Using the Google Cloud Console:** 1. Open the **Google Cloud Console**. 2. Navigate to **Storage → Buckets** and select the bucket you've configured for custom checkpoints. 3. Click **Upload folder** (or **Upload files**, depending on your structure). 4. Select the directory containing your converted checkpoint artifacts. 5. Wait for the upload to complete; the structure should remain intact. **Using the gcloud CLI:** You can upload the entire converted checkpoint directory recursively with: ```bash theme={null} gcloud storage cp -r gs://// ``` Register your checkpoint by creating a Model Manifest. The Model Manifest stores relevant information about your checkpoint such as the name to use in API requests, supported languages, and a description of the checkpoint. ### Model Manifest structure Below is an example Model Manifest: ```yaml theme={null} apiVersion: sambanova.ai/v1 kind: Model metadata: name: my-custom-llama3.1-8b # kubernetes resource name. Note that this spec: name: my-custom-llama3.1-8b # serving name used in the inference API owner: jane@doe.ai public: true aliases: - my-custom-llama3.1-8b # other aliases for the serving name used in the inference API - My-Custom-Llama3.1-8B metadata: name: "[optional]" # kubernetes name overview: "[optional description of the model]" architecture: "[optional]" category: "[optional]" status: active github_link: "[optional]" hf_link: "[optional]" languages: - "[optional]" license: "[optional]" vocabulary_size: "[optional]" price: input_tokens: "[optional]" output_tokens: "[optional]" tokenizer: endpointUrl: "" # optional remote tokenizer endpoint path: ./Meta-Llama-3.1-8B-Instruct_tokenizer ``` The `name` field (e.g., `My-Custom-Llama3.1-8B`) will be the name you use in subsequent steps and in API requests. ### Configuring the tokenizer field The `tokenizer` field can be set to the base model used for your custom checkpoint. For instance, if your custom checkpoint is fine-tuned from `Meta-Llama-3.1-70B-Instruct`, set the tokenizer path as follows: ```yaml theme={null} tokenizer: endpointUrl: '' path: ./Meta-Llama-3.1-70B-Instruct_tokenizer ``` The tokenizer field in the Model Manifest is **only** used for running checks on the inputs to calculate sequence length requirements prior to generation time. ### Applying the Model Manifest After creating your Model Manifest, apply it using: ```bash theme={null} kubectl apply -f your_model_manifest_filename.yaml ``` Deploying a custom checkpoint follows the same overall workflow as [deploying any standard model in SambaStack](/en/v1.2.0/sambastack/service-administration/deploying-model-bundles). The most straightforward approach is to start from an existing **Bundle Template** that uses the **same model architecture** as your custom checkpoint and then modify the relevant fields to point to your custom artifacts. If you are unfamiliar with Bundle Templates, model deployment, or how to choose a template for your use-case, see the [Deploying Bundles](/en/v1.2.0/sambastack/service-administration/deploying-model-bundles) page. ### Understanding Bundle structure The objects in the Bundle definition to modify are: 1. **`checkpoints`** – defines checkpoint aliases and their GCS source paths 2. **`models`** – maps model names to checkpoints and templates Below is an example Bundle for the base `Meta-Llama-3.1-70B-Instruct` model: ```yaml theme={null} apiVersion: sambanova.ai/v1alpha1 kind: Bundle metadata: name: 8b-3dot1-full spec: checkpoints: LLAMA3_8B_3_1_CKPT: source: gs://path/to/ckpt/in/gcs # gs:// -> Google Cloud Storage toolSupport: true LLAMA3_70B_3_1_CKPT: source: nfs:///nfsdata/path/to/ckpt/in/nfs # nfs:// -> NFS path mounted in the serving pod models: Meta-Llama-3.1-8B-Instruct: checkpoint: LLAMA3_8B_3_1_CKPT template: Meta-Llama-3.1-8B-Instruct Meta-Llama-3.1-70B-Instruct: checkpoint: LLAMA3_70B_3_1_CKPT template: Meta-Llama-3.1-70B-Instruct secretNames: - sambanova-artifact-reader template: 8b-3dot1-full ``` ### Updating the checkpoints section For`spec.checkpoints`: * Change the checkpoint key to a name (alias) for your custom checkpoint * Update the `source` field to the GCS path of your converted checkpoint directory * Set `toolSupport` to `true` or `false` depending on whether your fine-tuned checkpoint is configured for tool-use For example: ```yaml theme={null} checkpoints: CUSTOM_CHECKPOINT_ALIAS: source: nfs:///nfsdata/path/to/custom/ckpt/dir toolSupport: false ``` `spec.checkpoints.source` can either be a google cloud storage path (`gs://`) or nfs (`nfs://`). In the above example, `/nfsdata` is the path where NFS is mounted in the serving pod. ### Updating the models section For `spec.models`: * Change the model key to **the name you defined in the Model Manifest** (this is the name you will use in inference API calls) * Set the `checkpoint` field to the alias you defined above * Leave `template` as the original Model Template For example: ```yaml theme={null} models: My-Custom-Llama3.1-8B: checkpoint: CUSTOM_CHECKPOINT_ALIAS template: Meta-Llama-3.1-8B-Instruct ``` The `template` field should reflect the base model. For instance, if your custom checkpoint is fine-tuned from `Meta-Llama-3.3-70B-Instruct`, then the template field should be `Meta-Llama-3.3-70B-Instruct`. ### Applying the Bundle Once you've updated the Bundle configuration, apply it: ```bash theme={null} kubectl apply -f your_bundle_filename.yaml ``` After your deployment is running, use the model name you defined (e.g., `My-Custom-Llama3.1-8B`) in your inference API requests. ## Verifying your deployment After applying the Bundle, verify that your custom checkpoint deployment is successful: 1. **Check deployment status:** ```bash theme={null} kubectl get bundles kubectl describe bundle ``` 2. **Verify the model is available:** ```bash theme={null} kubectl get models ``` 3. **Test with a sample inference request** using your custom model name. See the [Quickstart Guide for Developers](/en/get-started/quickstart) for example inference requests using the SambaNova SDK, OpenAI-compatible libraries, or CURL. ## Troubleshooting ### Common issues | Issue | Possible Cause | Solution | | -------------------------------------- | ----------------------------------------------- | ------------------------------------------------------------------------------------- | | Deployment fails with permission error | Service account lacks read access to GCS bucket | Verify Storage Object Viewer role is granted (see Step 2) | | Model not found in API requests | Model name mismatch between Manifest and Bundle | Ensure the model name in the `models` section matches the Model Manifest `name` field | | Checkpoint files not found | Incorrect GCS path in Bundle | Verify the `source` path matches your uploaded checkpoint location | | Inference errors | Checkpoint incompatible with template | Ensure your custom checkpoint uses the same architecture as the specified template | ### Verifying GCS access If you suspect permission issues, verify that your service account can access the checkpoint: ```bash theme={null} gcloud auth activate-service-account --key-file= gsutil ls gs://// ``` ## Next steps * To deploy custom checkpoints with speculative decoding, see [Deploying with speculative decoding](/en/v1.2.0/sambastack/service-administration/deploy-with-speculative-decoding) * For monitoring and observability, see [SambaStack Monitoring](/en/v1.2.0/sambastack/reference-architecture/observability/overview) * If your custom checkpoint uses a different chat template or tool-call output format than the base model, see [Custom chat templates and output parsing](/en/build/chat-templates) in the Developer Guide for how to handle prompt formatting and parsing on the client side.