> ## Documentation Index
> Fetch the complete documentation index at: https://sambanova-systems.mintlify.site/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Deploying models and bundles

SambaStack manages model serving through a set of Kubernetes custom resources. This page covers how those resources relate to one another, how to inspect the ones already present in your cluster, and which workflow to follow for the deployment you want. The pages in this section walk through each workflow; the field reference for an individual resource lives on its own page under **Custom resources**.

In SambaStack, a deployment serves one or more models on a single node, together with their configurations, including batch sizes and sequence lengths. This approach uses the SambaNova Reconfigurable Dataflow Unit (RDU) to support multiple models and configurations in a single deployment. Because every model in a deployment is already resident, switching between them does not require reloading weights, which increases throughput compared with deployments that load a single static model.

For example, a single bundle can serve both `Llama-3.3-70B-Instruct` and `Llama-3.1-8B-Instruct`, allowing you to switch between them almost instantly. Each configuration occupies space on the node, so different bundles contain different sets of configurations. A copy of `Llama-3.3-70B-Instruct` with a batch size of 4 and a sequence length of 16k represents one configuration. For the bundles that SambaNova provides, see [Supported models and bundles](/docs/en/sambastack/service-administration/model-deployment/supported-models-and-bundles).

<Note>
  SambaStack v2.0.2 introduces a new set of Kubernetes custom resources (CRs) to manage how bundles are deployed. The CRs in older versions of SambaStack are deprecated; however, they remain functional until September 30, 2026. Model bundles built using the deprecated CRs [will need to be migrated to the new resource model](/docs/en/sambastack/service-administration/model-deployment/deploying-models-and-bundles/sambastack-v2-bundle-migration).
</Note>

<Note>
  A bundle is optional. To serve a single model, pair it with a model profile and deploy it directly. Create a bundle when you want to serve several models as one unit, configure speculative decoding, or validate and share a named configuration.
</Note>

## Terminology

| Term                     | Definition                                                                                                                                                                   |
| ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **RDU**                  | Reconfigurable Dataflow Unit - SambaNova's proprietary processor architecture                                                                                                |
| **PEF**                  | Processor Executable Format - Compiled model binaries that run on RDUs                                                                                                       |
| **Bundle**               | A deployable package combining models, their profiles, and their configurations                                                                                              |
| **Configuration**        | The runtime settings for a model, including batch size and sequence length. A single deployment can include multiple configurations, enabling instant switching between them |
| **Model architecture**   | The `model_arch` value shared by a set of PEFs, which determines the checkpoints those PEFs are compatible with                                                              |
| **Expert**               | A sequence length profile configuration (for example, 8k, 16k, 32k) within a model                                                                                           |
| **Tier**                 | A sequence length key in a batching configuration (for example, `8k` or `128k`), which maps to the batch sizes served at that length                                         |
| **Speculative Decoding** | An optimization technique using a smaller draft model to accelerate inference from a larger target model                                                                     |
| **Legalizer**            | A validation process that verifies a configuration fits within RDU memory constraints                                                                                        |
| **CR (Custom Resource)** | A Kubernetes extension that defines custom resource types such as `Pef`, `Model`, `ModelProfile`, `ModelDeployment`, and `ModelBundle`                                       |

## Resource architecture

This section covers the resource structure used when creating **custom** deployments. All of these resources use `apiVersion: sambanova.ai/v1alpha1`.

```mermaid theme={}
graph TD
    accTitle: How the SambaStack model deployment custom resources relate to one another
    accDescr: A PEF is referenced by a ModelProfile. A ModelProfile and a Model form an implicit pair, which is not a custom resource. That pair is referenced by a ModelBundle, and the bundle is served by a ModelDeployment. The pair can also be deployed directly by a ModelDeployment without a bundle. PEF, ModelProfile, and Model are provided with the SambaStack installation. ModelBundle and ModelDeployment are the resources you author.

    PEF["PEF<br/><small>provided&nbsp;·&nbsp;references to versioned executables</small>"]
    MP["ModelProfile<br/><small>provided&nbsp;·&nbsp;set&nbsp;of&nbsp;feature‑compatible&nbsp;PEFs&nbsp;for&nbsp;a&nbsp;model&nbsp;arch<br/>+&nbsp;named&nbsp;batching&nbsp;configurations</small>"]
    M["Model<br/><small>provided&nbsp;·&nbsp;checkpoints per architecture</small>"]
    MB["ModelBundle<br/><small>you&nbsp;author&nbsp;·&nbsp;reusable,&nbsp;shareable&nbsp;set&nbsp;of<br/>models&nbsp;+&nbsp;profiles</small>"]
    MD["ModelDeployment<br/><small>you author&nbsp;·&nbsp;routable inference endpoint, replicas &amp; QoS, backed by serving pods</small>"]
    JP("<i>ModelProfile & Model Pair</i><br/><small>not a CR&nbsp;·&nbsp;implicit pairing<br/>model_arch must be compatible</small>")

    PEF -->|"referenced by"| MP
    MP --- JP
    M --- JP
    JP -->|"referenced by"| MB
    MB ==>|"bundle deploy"| MD
    JP -.->|"direct deploy"| MD

    classDef cluster fill:#eef2f7,stroke:#8aa0bd,color:#1a2b45,stroke-dasharray:4 3;
    classDef authored fill:#cfe2f3,stroke:#2f6fb0,color:#0d2c4d,stroke-width:2px;
    classDef config fill:#fbe7c6,stroke:#c9871f,color:#5c3d00;
    class PEF,MP,M cluster;
    class MB,MD authored;
    class JP config;
```

Every node is tagged with its role. Nodes tagged *provided* have dashed borders; they come with the SambaStack installation and you reference them by name. Nodes tagged *you author* have solid borders; they are the resources you write. The rounded node tagged *not a CR* is not a custom resource at all, but a single `Model` and `ModelProfile` pairing, which corresponds to one entry in `spec.modelConfigs`.

The resources divide responsibility as follows:

1. **Pef** - Registers a compiled executable and the metadata describing what it supports, including its model architecture, batch size, and maximum sequence length.
2. **ModelProfile** - Defines *how* a model architecture can be run: which PEFs it uses, which features it supports, and the batch sizes available at each sequence length tier.
3. **Model** - Defines *which checkpoints* are available for each architecture, along with the tokenizer to use.
4. **ModelBundle** - Combines one or more model and profile pairs into a single named, deployable unit. Optional.
5. **ModelDeployment** - Instantiates one or more replicas on the cluster, either from a bundle or from an inline model and profile pair.

This separation allows you to:

* Deploy a single model by pairing it with a profile, without creating a bundle
* Reuse one profile across every model and checkpoint that shares its architecture, including custom checkpoints for fine-tuned models
* Serve a custom checkpoint by adding a `Model` resource only, leaving the profile untouched
* Group several models into one validated, shareable unit when you need them served together
* Update checkpoints without modifying profiles or deployment configurations
* Deploy the same configuration with different replica counts

<Note>
  Checkpoint paths are never declared in a bundle or a deployment. The operator resolves them from the `Model` you reference, either through the `<model-name>[:<arch>][:<version>]` reference or through `modelSettings.checkpointOverrides`.

  Compatibility between a checkpoint and a profile is defined by `model_arch`, but the operator does not verify it. Select a checkpoint whose architecture is compatible with the profile you reference.
</Note>

## Identify available models and profiles

Before authoring a deployment, identify the models and profiles available in your cluster and determine which pair together.

<Steps>
  <Step title="List available models">
    List the models and note the architecture keys under each model's `spec.checkpoints`:

    <Tabs>
      <Tab title="Hosted">
        ```bash theme={}
        kubectl get models
        ```

        Example:

        ```bash theme={}
        kubectl describe model gpt-oss-120b
        ```
      </Tab>

      <Tab title="On Premise">
        ```bash theme={}
        kubectl -n <namespace> get models.sambanova.ai
        ```

        Example:

        ```bash theme={}
        kubectl -n <namespace> describe model.sambanova.ai gpt-oss-120b
        ```
      </Tab>
    </Tabs>

    <Note>
      These commands list Kubernetes resource names (`metadata.name`), which are not the names you send to the inference API. Use the serving name from `spec.name` in requests. See [Supported models and bundles](/docs/en/sambastack/service-administration/model-deployment/supported-models-and-bundles#supported-models).
    </Note>
  </Step>

  <Step title="List available profiles">
    List the profiles and note the `model_arch` value of each:

    <Tabs>
      <Tab title="Hosted">
        ```bash theme={}
        kubectl get modelprofiles
        ```
      </Tab>

      <Tab title="On Premise">
        ```bash theme={}
        kubectl -n <namespace> get modelprofiles.sambanova.ai
        ```
      </Tab>
    </Tabs>

    Output:

    ```text theme={}
    deepseek              17h
    gpt-oss-fp8-dyt       17h
    llama-3p1-70b-sd      17h
    llama-3p1-8b          17h
    ```
  </Step>

  <Step title="Match a profile to a model">
    View a profile to confirm its architecture, features, and batching support:

    <Tabs>
      <Tab title="Hosted">
        ```bash theme={}
        kubectl describe modelprofile <profile-name>
        ```
      </Tab>

      <Tab title="On Premise">
        ```bash theme={}
        kubectl -n <namespace> describe modelprofile.sambanova.ai <profile-name>
        ```
      </Tab>
    </Tabs>

    A profile can be used with a model when the profile's `model_arch` value matches one of the architecture keys under the model's `spec.checkpoints`. The operator does not verify this pairing, so confirm it yourself before deploying. Review the profile for:

    * `model_arch` - The architecture the profile targets
    * `features` - Optional features the profile's PEFs support
    * `batchingConfigs` - Named batching configurations, each mapping sequence length tiers to their batch sizes
    * `pefs` - The PEF versions the profile references
  </Step>
</Steps>

<Tip>
  A profile whose `features` list includes `speculative_decoding` requires a draft model. Pair it using `spec.specDecodingPairs` in a bundle.
</Tip>

## Choose a workflow

<CardGroup cols={2}>
  <Card icon="rocket" href="/docs/en/sambastack/service-administration/model-deployment/deploying-models-and-bundles/deploy-a-single-model" title="Deploy a single model">
    Pair a model with a profile and deploy it directly, without a bundle
  </Card>

  <Card icon="layer-group" href="/docs/en/sambastack/service-administration/model-deployment/deploying-models-and-bundles/deploy-a-bundle" title="Deploy a bundle">
    Serve a bundle that SambaNova provides, or one you authored
  </Card>

  <Card icon="wrench" href="/docs/en/sambastack/service-administration/model-deployment/deploying-models-and-bundles/create-a-custom-bundle" title="Create a custom bundle">
    Combine several models and profiles into one named, validated unit
  </Card>

  <Card icon="compress" href="/docs/en/sambastack/service-administration/model-deployment/deploying-models-and-bundles/deploy-custom-checkpoints" title="Deploying custom checkpoints">
    Convert, upload, and serve your own custom or fine-tuned checkpoint
  </Card>

  <Card icon="gauge-high" href="/docs/en/sambastack/service-administration/model-deployment/deploying-models-and-bundles/deployment-configurations" title="High-throughput vs. high-interactivity">
    Choose between the two dedicated deployment configurations
  </Card>

  <Card icon="arrow-right-arrow-left" href="/docs/en/sambastack/service-administration/model-deployment/deploying-models-and-bundles/sambastack-v2-bundle-migration" title="Migrating bundle configuration">
    Migrate bundles built with the deprecated custom resources
  </Card>
</CardGroup>