Build with LiteLLM - SambaNova Integration Guide

This article shows you how to use LiteLLM with SambaNova to make your first completion call. LiteLLM is an open-source Python library that provides a unified interface for accessing LLMs, translating inputs and mapping exceptions. LiteLLM ships a native sambanova provider, so you address SambaCloud models as sambanova/<model-id> and LiteLLM supplies the endpoint for you. You do not need to set a base URL.

Prerequisites

Before you begin, ensure you have:

A SambaCloud account with an active API key
Python 3.10 to 3.14. litellm does not support Python 3.9 or earlier, or Python 3.15 or later.
A model ID from the SambaCloud models page. Every model listed there works with the sambanova/ prefix.

macOS ships with Python 3.9 and exposes it as python3 rather than python. Check your version with python3 --version. If it is below 3.10, install a supported version with pyenv, uv, or brew install python@3.12 before continuing.

Installation

Create and activate a virtual environment, then install LiteLLM:

python3 -m venv .venv
source .venv/bin/activate
python --version
pip install litellm

The examples on this page were written against litellm 1.96.1. To pin that exact version, run pip install 'litellm==1.96.1' instead.

Example code

Export your API key, so it stays out of your source files:

export SAMBANOVA_API_KEY="your-api-key"

Then make your first completion call. LiteLLM reads the key from the environment. Save the following as first_call.py:

from litellm import completion

response = completion(
    model="sambanova/Meta-Llama-3.3-70B-Instruct",
    messages=[{"role": "user", "content": "What is SambaNova?"}],
)
print(response.choices[0].message.content)

Run it:

python first_call.py

You should see output similar to the following.

SambaNova Systems builds AI hardware and software for running large language
models. Its SambaCloud service provides API access to open source models.

completion() returns a ModelResponse. The answer is a plain string at response.choices[0].message.content, and the response also carries id, model, created, and choices[0].finish_reason if you need them. To use a different model, change the string after sambanova/ to any ID from the SambaCloud models page. Nothing else in the call changes.

Advanced options

LiteLLM passes SambaNova’s request parameters through to the API. The following requests a JSON-formatted response with token and sampling controls:

from litellm import completion

response = completion(
    model="sambanova/Meta-Llama-3.3-70B-Instruct",
    messages=[
        {
            "role": "user",
            "content": "What do you know about sambanova.ai? Give your response in JSON format.",
        }
    ],
    max_tokens=512,
    response_format={"type": "json_object"},
    temperature=0.2,
    top_p=0.9,
)
print(response.choices[0].message.content)

With response_format={"type": "json_object"}, content is still a string containing JSON, not a dictionary. Parse it before use, reusing the response from the previous example:

import json

data = json.loads(response.choices[0].message.content)
print(data.keys())

JSON mode guarantees the response parses, not which keys it contains. The model picks those from your prompt, so inspect the keys before depending on them, or state the exact schema you want in the prompt.

Stream the response

Pass stream=True to receive tokens as the model produces them. Streaming returns an iterator of chunks instead of a single ModelResponse, and the text arrives at chunk.choices[0].delta.content. That field is None on the final chunk, so the or "" below keeps print() from failing at the end of the stream:

from litellm import completion

stream = completion(
    model="sambanova/Meta-Llama-3.3-70B-Instruct",
    messages=[{"role": "user", "content": "What is SambaNova?"}],
    stream=True,
)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="", flush=True)
print()

Track the cost of a call

litellm.completion_cost() returns the cost of a response in US dollars, calculated from the price table bundled with LiteLLM:

import litellm
from litellm import completion

response = completion(
    model="sambanova/Meta-Llama-3.3-70B-Instruct",
    messages=[{"role": "user", "content": "What is SambaNova?"}],
)
print(response.usage.prompt_tokens, response.usage.completion_tokens)
print(litellm.completion_cost(completion_response=response))

LiteLLM’s price table is maintained by the LiteLLM project, not by SambaNova, and can lag a pricing change. Treat the SambaCloud pricing page as authoritative for billing.

Route across models with fallbacks

Passing fallbacks makes LiteLLM retry a failed request against other models instead of raising. LiteLLM tries model first, then each entry in fallbacks in order, and returns the first response that succeeds. If every attempt fails, it raises the last error:

from litellm import completion

response = completion(
    model="sambanova/Meta-Llama-3.3-70B-Instruct",
    messages=[{"role": "user", "content": "What is SambaNova?"}],
    fallbacks=["sambanova/gpt-oss-120b", "sambanova/DeepSeek-V3.1"],
)
print(response.model)
print(response.choices[0].message.content)

response.model reports which model produced the answer. Because LiteLLM resolves each entry independently, a fallback list can also cross providers, as long as you have set that provider’s API key in the environment.

Run the LiteLLM proxy

The proxy puts an OpenAI-compatible endpoint in front of your models, so any OpenAI client can reach SambaCloud without code changes. Install the proxy dependencies:

pip install 'litellm[proxy]'

Create config.yaml. The model_name is the alias your clients request, and litellm_params.model is the real SambaNova model. Use the os.environ/ prefix so the key is read from the environment rather than stored in the file:

model_list:
  - model_name: llama-3.3-70b
    litellm_params:
      model: sambanova/Meta-Llama-3.3-70B-Instruct
      api_key: os.environ/SAMBANOVA_API_KEY
  - model_name: gpt-oss-120b
    litellm_params:
      model: sambanova/gpt-oss-120b
      api_key: os.environ/SAMBANOVA_API_KEY

Start the proxy, which listens on port 4000 by default:

litellm --config config.yaml

Then send a request using the alias you defined, not the SambaNova model ID:

curl http://0.0.0.0:4000/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3.3-70b",
    "messages": [{"role": "user", "content": "What is SambaNova?"}]
  }'

To self-host the proxy against SambaStack rather than SambaCloud, see the LiteLLM reference architecture page, which covers the database and deployment setup.

Troubleshooting

command not found: python

macOS does not provide a python executable, only python3. Use python3 -m venv .venv to create the environment. After you activate it with source .venv/bin/activate, python works as expected inside the environment.

AuthenticationError: SambanovaException - Incorrect API key provided

SAMBANOVA_API_KEY is unset or wrong. LiteLLM reads it from the environment at call time, so this surfaces on the completion() call rather than at import. Confirm with echo $SAMBANOVA_API_KEY, and note that an export applies only to the shell it ran in.

BadRequestError: LLM Provider NOT provided

The sambanova/ prefix is missing from the model name. LiteLLM selects the provider from that prefix, so the model must be written as sambanova/Meta-Llama-3.3-70B-Instruct, not Meta-Llama-3.3-70B-Instruct.

NotFoundError: The model does not exist or you do not have access to it

The model ID after sambanova/ is misspelled or no longer served. Model IDs are case-sensitive, so sambanova/meta-llama-3.3-70b-instruct fails where sambanova/Meta-Llama-3.3-70B-Instruct succeeds. Check the ID against the SambaCloud models page, and check the model deprecations page for a replacement if the model has been retired.

The model returns prose instead of JSON

response_format={"type": "json_object"} constrains the output format, but the request should also ask for JSON in the prompt itself, as the advanced example does. Confirm the model you chose supports structured output on the SambaCloud models page.

The proxy fails to start with a missing module error

pip install litellm installs the litellm command but not the server dependencies the proxy needs, such as uvicorn and fastapi. Run pip install 'litellm[proxy]' inside your activated virtual environment. Quote the argument, because zsh treats unquoted square brackets as a glob pattern.

The proxy returns a model not found error for a name that works in Python

Requests to the proxy use the model_name alias from config.yaml, not the sambanova/<model-id> string. Send "model": "llama-3.3-70b" if that is the alias you defined, and keep the sambanova/ prefix only in litellm_params.model.

Additional resources

LiteLLM SambaNova provider documentation
LiteLLM proxy documentation
SambaCloud models for the current model IDs

Overview

Developer tools

Frameworks

Interfaces and low-code

Data and infrastructure

Multimodal

Evaluation and monitoring

LiteLLM integration guide

Prerequisites

Installation

Example code

Advanced options

Stream the response

Track the cost of a call

Route across models with fallbacks

Run the LiteLLM proxy

Troubleshooting

Additional resources

​Prerequisites

​Installation

​Example code

​Advanced options

​Stream the response

​Track the cost of a call

​Route across models with fallbacks

​Run the LiteLLM proxy

​Troubleshooting

​Additional resources

Prerequisites

Installation

Example code

Advanced options

Stream the response

Track the cost of a call

Route across models with fallbacks

Run the LiteLLM proxy

Troubleshooting

Additional resources