Deploy a Single Model on SambaStack - SambaNova Documentation

To serve one model, pair it with a profile and deploy it directly with a ModelDeployment. No bundle is created. Create a bundle instead when you want to serve several models as one unit, configure speculative decoding, or validate and share a named configuration. See Create a custom bundle.

Prerequisites

Before deploying, complete the quickstart that applies to you:

Quickstart - Hosted

System set up for hosted SambaStack

Quickstart - On-prem

System set up for on-prem SambaStack

Additionally:

Review the deployment overview for how models, profiles, and deployments fit together.
Identify the model and profile you want to pair, and confirm they are compatible. See Identify available models and profiles.
Check Supported models and bundles for the available model checkpoints.

Deploy the model

Create the YAML file

Create a YAML file that declares the model and profile inline under spec.models:

apiVersion: sambanova.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: md-gpt120
spec:
  models:
    modelConfigs:
    - model: gpt-oss-120b:1
      profile: gpt-oss-fp8-dyt
  groups:
  - minReplicas: 1
    name: default
    qosList:
    - free
  owner: no-reply@sambanova.ai
  secretNames:
  - sambanova-artifact-reader
  engineConfig:
    startupTimeout: 7200

A checkpoint path is not required, because the checkpoint is resolved from the referenced Model.startupTimeout is in seconds. The 7200 here gives the engine two hours to download and load the model before its startup probe fails, well above the 45-minute default. See Startup timeouts.

Apply the ModelDeployment

Hosted
On Premise

kubectl apply -f <modeldeployment-file>.yaml

kubectl -n <namespace> apply -f <modeldeployment-file>.yaml

Monitor deployment status

Hosted
On Premise

kubectl get modeldeployments
kubectl describe modeldeployment <deployment-name>

kubectl -n <namespace> get modeldeployments.sambanova.ai
kubectl -n <namespace> describe modeldeployment.sambanova.ai <deployment-name>

The deployment is complete once all of its serving pods report a ready status.

Verify the deployment

Pods reporting ready means the serving containers started, not that the model answers requests. Confirm the serving name, then send one inference request.

Resolve the serving name

The name you send to the API is not the Kubernetes resource name. Three names are involved:

Name	Where it lives	Used for
`metadata.name` on a Model	The `Model` resource	`kubectl` commands, and `modelConfigs[].model` references in bundles and deployments
`spec.name` on a Model	The `Model` resource	The `model` field of an inference request
`spec.aliases` on a Model	The `Model` resource	Additional names that route to the same model

Read the serving name from the Model you deployed:

Hosted
On Premise

kubectl get model <model-resource-name> -o jsonpath='{.spec.name}{"\n"}'

kubectl -n <namespace> get model.sambanova.ai <model-resource-name> -o jsonpath='{.spec.name}{"\n"}'

Then confirm the gateway is serving it. Each id in the response is a servable name:

curl https://<your-api-domain>/v1/models \
  -H "Authorization: Bearer $SAMBANOVA_API_KEY"

A model set to modelSettings.routable: false, such as a speculative decoding draft model, is deliberately absent from this list and cannot be addressed directly.

Send a request

Use the serving name in the model field:

curl https://<your-api-domain>/v1/chat/completions \
  -H "Authorization: Bearer $SAMBANOVA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-oss-120b",
    "messages": [
      {"role": "system", "content": "You are a concise assistant."},
      {"role": "user", "content": "Say hi in one sentence."}
    ],
    "max_tokens": 128,
    "stream": false
  }'

A successful response returns the serving name in model and a populated choices[0].message:

{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": "Hi there, it's a sunny day today!",
        "role": "assistant"
      }
    }
  ],
  "created": 1770927918,
  "id": "c754b468-05e1-4fe4-8a28-f51ecc74f4f2",
  "model": "gpt-oss-120b",
  "object": "chat.completion",
  "system_fingerprint": "fastcoe",
  "usage": {
    "completion_tokens": 121,
    "prompt_tokens": 91,
    "total_tokens": 212,
    "stop_reason": "stop",
    "time_to_first_token": 0.05286884307861328,
    "total_latency": 0.29257988929748535
  }
}

The usage object carries additional throughput and latency fields; for the full body, and for the API domain to use in each environment, see Quickstart - Hosted or Quickstart - On-prem.

A model not found error with pods running usually means the request used metadata.name instead of spec.name. An inference error on a model that loads and lists usually means the checkpoint and the profile disagree on model_arch. See Deployment failures.

Update or remove the deployment

To change a deployment’s configuration, edit its YAML file and reapply it. The legalizer automatically revalidates the changes.

Hosted
On Premise

kubectl apply -f <modified-file>.yaml

kubectl -n <namespace> apply -f <modified-file>.yaml

To remove the deployment:

Hosted
On Premise

kubectl delete modeldeployment <deployment-name>

kubectl -n <namespace> delete modeldeployment <deployment-name>

Troubleshooting

A deployment created from an inline spec.models specification is legalized in place, so both classes of failure can apply to it. See Troubleshooting deployments for legalizer validation failures and deployment failures.

ModelDeployment

Full field reference for the deployment resource

Deploy a bundle

Serve several models together as one unit

Getting Started

Service Administration

Hardware Administration

Reference Architecture

Resources

Deploy a single model

Prerequisites

Quickstart - Hosted

Quickstart - On-prem

Deploy the model

Verify the deployment

Resolve the serving name

Send a request

Update or remove the deployment

Troubleshooting

ModelDeployment

Deploy a bundle

​Prerequisites

Quickstart - Hosted

Quickstart - On-prem

​Deploy the model

​Verify the deployment

​Resolve the serving name

​Send a request

​Update or remove the deployment

​Troubleshooting

​Related documentation

ModelDeployment

Deploy a bundle

Prerequisites

Deploy the model

Verify the deployment

Resolve the serving name

Send a request

Update or remove the deployment

Troubleshooting

Related documentation